Tools that have been useful to me in the past, or might be useful in the future, loosely grouped. Not things I use daily.


🦋 Web scrapers

Curl impersonate

A special build of curl that can impersonate the four major browsers: Chrome, Edge, Safari & Firefox. curl-impersonate is able to perform TLS and HTTP handshakes that are identical to that of a real browser.

When you use an HTTP client with a TLS website, it first performs a TLS handshake. The first message of that handshake is called Client Hello. The Client Hello message that most HTTP clients and libraries produce differs drastically from that of a real browser.

If the server uses HTTP/2, then in addition to the TLS handshake there is also an HTTP/2 handshake where various settings are exchanged. The settings that most HTTP clients and libraries use differ as well from those of any real browsers.

For these reasons, some web services use the TLS and HTTP handshakes to fingerprint which client is accessing them, and then present different content for different clients. These methods are known as TLS fingerprinting and HTTP/2 fingerprinting respectively. Their widespread use has led to the web becoming less open, less private and much more restrictive towards specific web clients

With the modified curl in this repository, the TLS and HTTP handshakes look exactly like those of a real browser.

Puppeteer heap snapshot

puppeteer-heap-snapshot is a Node.js module that, given a Puppeteer browser page, can capture and parse a heap snapshot and deserialize objects that contain a set of properties. It comes with a nifty CLI tool too so we can quickly prototype scrapers from our terminal.

Instead of trying to use CSS selectors to get at the data we want, we grab the data straight out of the browser’s working memory.

Archive Team’s grab-site

grab-site is an easy preconfigured web crawler designed for backing up websites. Give grab-site a URL and it will recursively crawl the site and write WARC files. Internally, grab-site uses a fork of wpull for crawling.

Note

Still looking for something that can maintain an archive of a Facebook group.


🐌 Convert raster to vector (unsorted)

https://www.autotracer.org/

https://www.vectorizer.io/

https://vectorizer.ai/

https://www.vectorization.org/

https://www.visioncortex.org/vtracer/

https://svgco.de/

https://svgconverter.app/

https://online.rapidresizer.com/tracer.php

https://fconvert.com/autotrace/

https://online-converting.com/autotrace/

https://github.com/fromtheexchange/image2svg-awesome


📈 Convert text to diagram

Mermaid

JavaScript based diagramming and charting tool that renders Markdown-inspired text definitions to create and modify diagrams dynamically.

GraphViz

Not the prettiest output, but the dot language makes generating large diagrams really simple. I use it whenever I need to parse structure (eg library dependencies) into a graphical overview.

PyGraphViz

Consumes dot files and lets you query them. What’s the shortest path between A and B? What relies on C?


🔪 Edit PDFs

Stirling PDF

This is absolutely fantastic. A whole bunch of PDF tools, slapped in a Docker container with a simple and effective web interface on top. Less than five minutes to get running. I used it to generate my John Pory PDF.


🦀 Shell scripting

Gum

A collection of useful utilities for enhancing shell scripts.

chezmoi

Dotfile (Unix config file) manager. Maybe it’s better than a github repo, maybe it’s just another yak waiting to be shaved.

direnv

Set environment variables based on path.


🦚 Graphics libraries

Rough.js

Rough.js is a small (<9kB gzipped) graphics library that lets you draw in a sketchy, hand-drawn-like, style. The library defines primitives to draw lines, curves, arcs, polygons, circles, and ellipses. It also supports drawing SVG paths.


Programming

Zed

Code editor. Clean. Fast. Configurable key bindings. I don’t use one tenth of the features of an IDE, so this might be perfect for me.


🌀 Other

Al Dente

Battery charge limiter for macs.

bucklespring

Makes your MBP sound like a clicky-clacky buckling spring keyboard. I know it sounds like a joke. It probably is a joke. But for some reason it really helps me focus.

Fire Toolbox

Enhance the capabilities of Amazon’s Fire tablets.