This nascent site will be a repository for thoughts too big for a tweet that I really have to stop bothering my friends with. It will likely focus on natural language processing and machine learning, but some other topics might creep in from time to time. I’ll be impressed if I manage two posts a year.

Some earlier posts are hosted on the Roam Analytics github page (previously on their website which no longer exists). A few highlights:

• Are categorical variables getting lost in your random forests?: demonstrating a limitation of common random forest implementations when faced with categorical data.
• Prescription-based prediction: predicting a medical provider’s specialty from their prescribing patterns.
• Translation and scaling invariance in regression models: correcting some misconceptions about feature normalization for linear models.

Posts

• 

  Tokenization for language modeling: Byte Pair Encoding vs Unigram Language Modeling

  Tokenizers used by the best-performing language models (Bert, GPT-2, etc.) poorly reflect the morphology of English text. I had hoped to use some quarantine time to design one that more closely aligns to relationships between wordforms. But Kaj Bostrom and Greg Durrett beat me to it and so this blog post materialized instead. I add some additional motivation, evaluate both methods against 'gold standard' tokenizations, and speculate about what might come next.

  Jul 9, 2020