<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Sayak Dattagupta</title><link>https://sayakdattagupta.com/</link><description>Sayak Dattagupta personal website</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Wed, 28 Oct 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://sayakdattagupta.com/index.xml" rel="self" type="application/rss+xml"/><item><title>Rewriting LibrExam's indexer (and most of the app)</title><link>https://sayakdattagupta.com/post/2026/10/28/rewriting-librexams-indexer-and-most-of-the-app/</link><pubDate>Wed, 28 Oct 2026 00:00:00 +0000</pubDate><guid>https://sayakdattagupta.com/post/2026/10/28/rewriting-librexams-indexer-and-most-of-the-app/</guid><description>&lt;p&gt;A few days ago, I pushed the first substantial set of changes to &lt;a href="https://github.com/sayakdattagupta/librexam"&gt;LibrExam&lt;/a&gt; in over a year. The bulk of it went into a single commit touching 56 files, which reworked the storage layer, the results page and most of the interface; a complete rewrite of the indexer in &lt;a href="https://github.com/sayakdattagupta/librexamtests"&gt;librexamtests&lt;/a&gt; followed shortly after.&lt;/p&gt;
&lt;p&gt;I have previously written about &lt;a href="https://sayakdattagupta.com/post/2025/09/08/using-github-repos-as-decentralised-content-contribution/"&gt;how LibrExam uses GitHub to distribute tests&lt;/a&gt;, along with a list of everything wrong with that approach. Most of that list has now been dealt with, and the fixes - along with a few problems I only discovered while making them - are documented here.&lt;/p&gt;
&lt;p&gt;I have omitted the interface changes; they are better seen than described.&lt;/p&gt;
&lt;h2 id="the-indexer"&gt;The indexer&lt;/h2&gt;
&lt;p&gt;The basic idea remains the same: authors tag their repos with the &lt;code&gt;librexam-test&lt;/code&gt; topic, and a GitHub Actions workflow in my repo periodically searches for them. Everything around that search, however, has been rewritten.&lt;/p&gt;
&lt;p&gt;The workflow now runs every 30 minutes, inside a &lt;code&gt;concurrency&lt;/code&gt; group, so that two runs can never overlap. It also authenticates with the &lt;code&gt;GITHUB_TOKEN&lt;/code&gt; that every workflow is given, which raises the search limit from 10 requests a minute to 30. Results are paged through 100 at a time, for at most 10 pages; GitHub does not return more than 1,000 results for a single search anyway, and LibrExam is (comfortably) nowhere near that number.&lt;/p&gt;
&lt;p&gt;For every repo it finds, the indexer then:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Skips it if it is a fork, archived, or listed in my &lt;code&gt;blocklist.json&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Fetches its &lt;code&gt;index.json&lt;/code&gt; from the repo’s actual default branch.&lt;/li&gt;
&lt;li&gt;Rejects any entry whose &lt;code&gt;url&lt;/code&gt; is not an &lt;code&gt;https://&lt;/code&gt; link.&lt;/li&gt;
&lt;li&gt;Downloads every test in that index and validates it (more on that below).&lt;/li&gt;
&lt;li&gt;Writes everything that survived into a single, flat &lt;code&gt;tests.json&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Each entry in &lt;code&gt;tests.json&lt;/code&gt; carries everything the browser needs to display a test, so the browser never has to look at the repo itself. A (trimmed) entry looks like this:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-json"&gt;{
 &amp;quot;name&amp;quot;: &amp;quot;LibrExam Sample Test&amp;quot;,
 &amp;quot;filename&amp;quot;: &amp;quot;test1.json&amp;quot;,
 &amp;quot;author&amp;quot;: &amp;quot;Sayak&amp;quot;,
 &amp;quot;duration&amp;quot;: 5400,
 &amp;quot;tags&amp;quot;: [&amp;quot;sample&amp;quot;, &amp;quot;physics&amp;quot;, &amp;quot;chemistry&amp;quot;, &amp;quot;mathematics&amp;quot;, &amp;quot;jee&amp;quot;],
 &amp;quot;url&amp;quot;: &amp;quot;https://raw.githubusercontent.com/librexam/testRepo/main/tests/test1.json&amp;quot;,
 &amp;quot;repo&amp;quot;: &amp;quot;librexam/testRepo&amp;quot;,
 &amp;quot;updated_at&amp;quot;: &amp;quot;2026-10-26T11:34:57Z&amp;quot;,
 &amp;quot;stargazers_count&amp;quot;: 2
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Writes are atomic: each file is first written to a &lt;code&gt;.tmp&lt;/code&gt; file, which is then renamed over the real one. A run that fails halfway through, therefore, can never leave a half-written index behind.&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-js"&gt;function writeJson(file, data) {
 fs.writeFileSync(`${file}.tmp`, JSON.stringify(data, null, 2) + &amp;quot;\n&amp;quot;);
 fs.renameSync(`${file}.tmp`, file);
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The indexer also refuses to overwrite a non-empty index with an empty one. If a search suddenly returns zero repos when the previous run found some, it is far more likely that GitHub is having a bad day than that every test repo vanished at once; the run simply fails, and the old index stays.&lt;/p&gt;
&lt;h2 id="committing-only-when-something-changes"&gt;Committing only when something changes&lt;/h2&gt;
&lt;p&gt;Previously, the index was rewritten with a fresh timestamp on every run, whether or not anything had changed. Now, a commit is only made if the index actually differs from the last one:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-sh"&gt;if ! git diff --cached --quiet; then
 git commit -m &amp;quot;Update test index&amp;quot;
else
 last=$(git log -1 --format=%ct)
 if [ $(( $(date +%s) - last )) -gt $(( 50 * 86400 )) ]; then
 git commit --allow-empty -m &amp;quot;Keepalive (index unchanged)&amp;quot;
 else
 exit 0
 fi
fi
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The second branch exists for a slightly odd reason: GitHub disables scheduled workflows in public repos after 60 days without any activity. If nothing has changed for 50 days, the workflow therefore makes an empty commit, purely to keep itself scheduled.&lt;/p&gt;
&lt;h2 id="one-request-per-visit"&gt;One request per visit&lt;/h2&gt;
&lt;p&gt;Previously, every visitor’s browser fetched the central list of repos, and then the &lt;code&gt;index.json&lt;/code&gt; of each repo in it. This is a textbook N+1 problem; with five hundred repos, it would have meant five hundred and one requests for every single visit. Since the indexer now does the crawling, the browser only needs to fetch &lt;code&gt;tests.json&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The old per-repo approach has been kept, but only as a fallback:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-js"&gt;const list = (await loadFromTestsIndex()) || (await loadFromRepos());
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;It now also uses each repo’s &lt;code&gt;default_branch&lt;/code&gt;, instead of assuming that every repo’s branch is called &lt;code&gt;main&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="falling-back-to-jsdelivr"&gt;Falling back to jsDelivr&lt;/h2&gt;
&lt;p&gt;Every file LibrExam loads still comes from &lt;code&gt;raw.githubusercontent.com&lt;/code&gt;. However, any such URL can be mechanically rewritten into a jsDelivr URL for the exact same file:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-js"&gt;function sourcesFor(url) {
 const m = url.match(
 /^https:\/\/raw\.githubusercontent\.com\/([^/]+)\/([^/]+)\/([^/]+)\/(.+)$/,
 );
 if (!m) return [url];
 const [, user, repo, ref, path] = m;
 const mirror = `https://cdn.jsdelivr.net/gh/${user}/${repo}@${ref}/${path}`;
 return preferMirror ? [mirror, url] : [url, mirror];
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Each source is given 10 seconds (through an &lt;code&gt;AbortController&lt;/code&gt;) before the next one is tried. If the mirror ends up answering, &lt;code&gt;preferMirror&lt;/code&gt; is set, and the mirror is asked first for the remainder of the session; there is little point in waiting ten seconds for an unresponsive GitHub on every single request. jsDelivr does have caching of its own, so the mirror may occasionally be slightly behind - which is still preferable to it not being there at all.&lt;/p&gt;
&lt;h2 id="validation"&gt;Validation&lt;/h2&gt;
&lt;p&gt;Every test now passes through a single &lt;code&gt;validateTest()&lt;/code&gt; function, which is the exact same code in both the indexer and the app. Problems are sorted into two kinds:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Errors&lt;/strong&gt; mean that a test cannot run at all: a missing title, a non-positive &lt;code&gt;duration&lt;/code&gt;, empty subjects or sections, marks that are not numbers, fewer than two options, an &lt;code&gt;SMCQ&lt;/code&gt; answer that is not an option index, a &lt;code&gt;NUM&lt;/code&gt; answer that is not a number, an unknown question type, and so on. The indexer drops these tests entirely.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Warnings&lt;/strong&gt; mean that a test runs, but something in it is wrong - usually an answer that points to an option which does not exist. The indexer keeps these, and the app asks before loading one.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;None of this happens silently, either. Every error and warning is recorded in a &lt;code&gt;problems&lt;/code&gt; array alongside the tests in &lt;code&gt;tests.json&lt;/code&gt;, so an author can see exactly why their test did not appear. At the time of writing, one of the two indexed tests has an answer pointing to a fifth option that does not exist.&lt;/p&gt;
&lt;p&gt;Together with the blocklist and the fork/archive filter, this goes some way towards addressing the concern of unmoderated content. It does not solve it; it does, however, make it manageable.&lt;/p&gt;
&lt;h2 id="is-the-marking-any-closer-to-jee-advanced"&gt;Is the marking any closer to JEE Advanced?&lt;/h2&gt;
&lt;p&gt;Somewhat. The old version did award partial marks for multiple-correct questions - one mark per correct option picked - but this was the only scheme available, it was buried inside the results page’s view code, and it counted a partially correct answer as simply “correct” in the statistics.&lt;/p&gt;
&lt;p&gt;Scoring now lives in its own module, partially correct answers have their own &lt;code&gt;partial&lt;/code&gt; status, and each section (or question, which overrides its section) can choose a &lt;code&gt;partial&lt;/code&gt; mode: &lt;code&gt;perOption&lt;/code&gt;, &lt;code&gt;proportional&lt;/code&gt; or &lt;code&gt;none&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;For an &lt;code&gt;MMCQ&lt;/code&gt;, picking even one wrong option results in &lt;code&gt;-negM&lt;/code&gt;, and picking every correct option results in &lt;code&gt;posM&lt;/code&gt;. If only some of the correct options are picked, the mode decides:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-js"&gt;const mode = q.partial ?? section.partial ?? &amp;quot;perOption&amp;quot;;
if (mode === &amp;quot;none&amp;quot;) return { status: &amp;quot;incorrect&amp;quot;, marks: 0 };
if (mode === &amp;quot;proportional&amp;quot;) {
 const marks = Math.round((posM * hits * 100) / correct.size) / 100;
 return { status: &amp;quot;partial&amp;quot;, marks };
}
return { status: &amp;quot;partial&amp;quot;, marks: hits };
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;perOption&lt;/code&gt; remains the default, since it is roughly what JEE Advanced does; &lt;code&gt;proportional&lt;/code&gt; awards that fraction of the full marks instead, and &lt;code&gt;none&lt;/code&gt; awards a flat zero.&lt;/p&gt;
&lt;p&gt;Numerical questions can also have a &lt;code&gt;tolerance&lt;/code&gt; now. Previously, an answer had to match exactly, so &lt;code&gt;1.414&lt;/code&gt; against an answer of &lt;code&gt;1.4142&lt;/code&gt; was simply marked wrong. The tolerance is padded internally by &lt;code&gt;1e-9&lt;/code&gt;, since &lt;code&gt;0.1 + 0.2 === 0.3&lt;/code&gt; evaluates to &lt;code&gt;false&lt;/code&gt; in JavaScript, and a student should not lose marks to floating-point arithmetic.&lt;/p&gt;
&lt;h2 id="storage"&gt;Storage&lt;/h2&gt;
&lt;p&gt;This is where most of those 56 files went.&lt;/p&gt;
&lt;p&gt;Previously, every attempt - paused or submitted - was kept in a single &lt;code&gt;localStorage&lt;/code&gt; key, &lt;code&gt;librexam_test_history&lt;/code&gt;, as a snapshot that included the entire test. Attempting the same test five times meant storing five copies of it (six, counting the current attempt, which was saved separately as well). The whole array was also serialised and written out again every time progress was saved. Browsers usually give a site around 5 MB of &lt;code&gt;localStorage&lt;/code&gt;, and the old code did not handle running out of it at all.&lt;/p&gt;
&lt;p&gt;Now, every test is hashed with cyrb53 - a small, fast, non-cryptographic 53-bit string hash - and stored exactly once, under &lt;code&gt;librexam_test_&amp;lt;hash&amp;gt;&lt;/code&gt;. An attempt, stored under &lt;code&gt;librexam_attempt_&amp;lt;id&amp;gt;&lt;/code&gt;, only keeps its answers, timings and a &lt;code&gt;testHash&lt;/code&gt; pointing back to its test. Collisions are not a realistic concern at this scale, and there is nobody to defend against here.&lt;/p&gt;
&lt;p&gt;Tests that are no longer referenced by any attempt are then garbage collected, in a mark-and-sweep fashion:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-js"&gt;export function removeUnusedTests() {
 const used = new Set();
 for (const key of keysWithPrefix(ATTEMPT_PREFIX))
 used.add(readJson(key)?.testHash);
 used.add(readJson(CURRENT_KEY)?.testHash);

 for (const key of keysWithPrefix(TEST_PREFIX)) {
 const hash = key.slice(TEST_PREFIX.length);
 if (!used.has(hash)) {
 localStorage.removeItem(key);
 testCache.delete(hash);
 }
 }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Old saves are migrated automatically the first time the new version is opened: each snapshot has its test hashed and split out, after which &lt;code&gt;librexam_test_history&lt;/code&gt; is deleted. If storage does still fill up, every progress save now goes through &lt;code&gt;safeSet()&lt;/code&gt;, which catches the failure and asks the user (once) to export a backup from &lt;em&gt;Your Data&lt;/em&gt; and delete some old attempts.&lt;/p&gt;
&lt;p&gt;While working on this, I also found that downloaded tests were being cached under their file name alone - &lt;code&gt;json-${filename}&lt;/code&gt;. Two different repos with a &lt;code&gt;test1.json&lt;/code&gt; each would, therefore, quietly overwrite each other; the sample repo’s test is, of course, named &lt;code&gt;test1.json&lt;/code&gt;. The cache key now includes the repo as well.&lt;/p&gt;
&lt;h2 id="working-offline"&gt;Working offline&lt;/h2&gt;
&lt;p&gt;LibrExam now registers a service worker. On install, it fetches the app’s &lt;code&gt;index.html&lt;/code&gt;, extracts every &lt;code&gt;src&lt;/code&gt; and &lt;code&gt;href&lt;/code&gt; from it with a regular expression, and caches all of them. (Parsing HTML with a regular expression is generally a bad idea; for a single file whose structure I control, it is an acceptable one.)&lt;/p&gt;
&lt;p&gt;After that:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Page loads&lt;/strong&gt; go to the network first, and fall back to the cached app when offline.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Anything under &lt;code&gt;/assets/&lt;/code&gt;&lt;/strong&gt; is served from the cache first. This is safe because Vite includes a content hash in the name of every file it builds; if a file changes, so does its name.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Old caches&lt;/strong&gt; are deleted whenever the cache name (currently &lt;code&gt;librexam-v1&lt;/code&gt;) changes.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The service worker only handles LibrExam’s own files, though, and question images are hosted elsewhere. To cover those, LibrExam creates an &lt;code&gt;Image()&lt;/code&gt; for every &lt;code&gt;image&lt;/code&gt; and &lt;code&gt;simage&lt;/code&gt; URL in a test the moment it is loaded, so the browser has downloaded (and cached) all of them before the test even begins. Losing one’s connection halfway through a test, then, no longer means losing the diagrams as well.&lt;/p&gt;
&lt;h2 id="what-is-still-missing"&gt;What is still missing&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Images still need to be hosted elsewhere and linked in. Bundling a test together with its own images remains a possibility, not a plan.&lt;/li&gt;
&lt;li&gt;Each repo’s &lt;code&gt;index.json&lt;/code&gt; is still written by hand.&lt;/li&gt;
&lt;li&gt;Only two repos are indexed at the moment.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If you would like to host tests of your own, an &lt;code&gt;index.json&lt;/code&gt; at the root of a repo (and a valid test or two) along with the &lt;code&gt;librexam-test&lt;/code&gt; topic are all it takes for them to show up.&lt;/p&gt;
&lt;p&gt;Happy testing.&lt;/p&gt;</description></item><item><title>A collection of math tournaments (and entrance tests)</title><link>https://sayakdattagupta.com/post/2026/06/18/a-collection-of-math-tournaments-and-entrance-tests/</link><pubDate>Thu, 18 Jun 2026 00:00:00 +0000</pubDate><guid>https://sayakdattagupta.com/post/2026/06/18/a-collection-of-math-tournaments-and-entrance-tests/</guid><description>&lt;p&gt;Preparing for the JEE, and specifically JEE Advanced, exposed me to some foreign mathematics problems. Usually, these problem sets were from university-level mathematics competitions for first- and second-year undergraduate students, competitions organized by universities for high school students, or simply problems from other entrance examinations.&lt;/p&gt;
&lt;p&gt;I have omitted Mathematical Olympiads and their respective national-level equivalents.&lt;/p&gt;
&lt;p&gt;A 10+2 level student with a background in the JEE or any equivalent entrance exam, who also has a decent amount of additional interest in the subject, can tackle a fair number of problems from here.&lt;/p&gt;
&lt;h2 id="are-they-really-useful-for-the-jee"&gt;Are they really useful for the JEE?&lt;/h2&gt;
&lt;p&gt;The ones oriented towards high school students kind of are (except the IMO, obviously). The difficulty of these varies from very easy to very hard. Obviously, the primary aim of solving these is not to get a boost in the JEE; one can refer to my &lt;a href="https://sayakdattagupta.com/resources/"&gt;resources&lt;/a&gt; list for that.&lt;/p&gt;
&lt;h2 id="the-list"&gt;The list&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://sites.google.com/usc.edu/scmathcompetition/competitions/competition-archive"&gt;The Southern California Math Competition (USC)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://berkeley.mt/"&gt;Berkeley Math Tournament&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://artsci.tamu.edu/mathematics/outreach/programs/high-school-contest/index-past-subject-exams.html"&gt;TAMU High School Math Competition&lt;/a&gt; (Only the &amp;lsquo;DE&amp;rsquo; and &amp;lsquo;EF&amp;rsquo; exams)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://web.ma.utexas.edu/users/rusin/Bennett/index.html"&gt;UT Austin Bennett&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.cmi.ac.in/admissions/syllabus.php"&gt;CMI BSc Entrance Test&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://jason-shi-f9dm.squarespace.com/archives#directory"&gt;Princeton PUMaC&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://admission.isical.ac.in/syllabus-qp.html"&gt;ISI (B Stat and B Math)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.maths.ox.ac.uk/study-here/undergraduate-study/maths-admissions-test"&gt;The Oxford MAT&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.stanfordmathtournament.org/past-tests/problems"&gt;Stanford Math Tournament&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://icmathscomp.org/past-papers"&gt;Imperial-Cambridge Mathematics Competition&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.physicsandmathstutor.com/admissions/step/"&gt;Cambridge STEP&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.hmmt.org/www/archive/problems"&gt;Harvard-MIT HMMT&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://math.vt.edu/undergrad-math/vtrmc.html"&gt;Virginia Tech VTRMC&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="bonus-difficult-territory"&gt;Bonus: Difficult territory&lt;/h3&gt;
&lt;p&gt;Though I confess my capability to solve the problems from here are very limited, I must still remain inclusive&amp;hellip;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://kskedlaya.org/putnam-archive/"&gt;The William Lowell Putnam&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://math.mit.edu/~yyao1/integrationbee.html"&gt;MIT Integration Bee&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://mathiitm.com/events/intbee-24-25/"&gt;IIT Madras Integration Bee&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.theintegralcup.in/"&gt;The Integral Cup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.imc-math.org.uk/"&gt;IMC&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The severe deficiency of East Asian content here can only be cured once entire papers (mostly entrance papers) from Taiwan, China, South Korea, Japan, etc. are officially translated and released to the public. Until then, one must make do with the occasional odd question that appears in a &lt;a href="https://www.youtube.com/c/blackpenredpen"&gt;YouTube video&lt;/a&gt; or pops up on the &lt;a href="https://math.stackexchange.com/"&gt;Mathematics Stack Exchange&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;If I missed anything else, it is simply because I am not aware of it OR I have missed it (in which case I would update the list later).&lt;/p&gt;
&lt;p&gt;Happy solving.&lt;/p&gt;</description></item><item><title>Using GitHub as a decentralised contribution, centralised distribution system</title><link>https://sayakdattagupta.com/post/2025/09/08/using-github-repos-as-decentralised-content-contribution/</link><pubDate>Mon, 08 Sep 2025 00:00:00 +0000</pubDate><guid>https://sayakdattagupta.com/post/2025/09/08/using-github-repos-as-decentralised-content-contribution/</guid><description>&lt;p&gt;If you are reading this, you are probably already aware of &lt;a href="https://github.com/sayakdattagupta/librexam"&gt;LibrExam&lt;/a&gt;. LibrExam, as of now, has been my most monumental achievement in software development — and yet, it has been on a two-month-long development hiatus.&lt;/p&gt;
&lt;p&gt;Apologies.&lt;/p&gt;
&lt;p&gt;Nevertheless, my project uses a (clunky) decentralised content contribution system that I would like to talk about here. It’s highly likely someone has already built something similar to - but much better than - this, but I could not care less. I only believe in homegrown technology.&lt;/p&gt;
&lt;p&gt;I have used GitHub to achieve this. This means that neither I nor the end-user has to shell out money or computing power for a server, though this also fundamentally makes the system centralised.&lt;/p&gt;
&lt;figure&gt;&lt;img src="https://sayakdattagupta.com/assets/mrincrediblegithub.png" alt="toeswhoknows" width="851" height="508" loading="lazy" decoding="async"&gt;&lt;/figure&gt;&lt;p class="caption"&gt;
This image was actually created to reflect an unironic sentiment. I have repurposed it.
&lt;/p&gt;
&lt;p&gt;Let me elucidate how I did it. (Does this already sound like a LinkedIn post?)&lt;/p&gt;
&lt;h2 id="three-kinds-of-json"&gt;Three kinds of JSON&lt;/h2&gt;
&lt;p&gt;Everything in this system is a JSON file sitting in a GitHub repo. No database, no backend, no server that I pay for. There are exactly three kinds of files.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;1. The test itself.&lt;/strong&gt; A test is split into subjects, subjects into sections, and sections into questions. Marking is defined per section - &lt;code&gt;posM&lt;/code&gt; for a correct answer, &lt;code&gt;negM&lt;/code&gt; for the penalty on a wrong one - which is how JEE-style papers work anyway. A (heavily trimmed) version of the sample test:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-json"&gt;{
 &amp;quot;title&amp;quot;: &amp;quot;Sample Test&amp;quot;,
 &amp;quot;duration&amp;quot;: 3600,
 &amp;quot;author&amp;quot;: &amp;quot;Sayak&amp;quot;,
 &amp;quot;subjects&amp;quot;: [
 {
 &amp;quot;name&amp;quot;: &amp;quot;Physics&amp;quot;,
 &amp;quot;sections&amp;quot;: [
 {
 &amp;quot;name&amp;quot;: &amp;quot;Section 1&amp;quot;,
 &amp;quot;posM&amp;quot;: 4,
 &amp;quot;negM&amp;quot;: 1,
 &amp;quot;questions&amp;quot;: [
 {
 &amp;quot;type&amp;quot;: &amp;quot;SMCQ&amp;quot;,
 &amp;quot;text&amp;quot;: &amp;quot;What is the standard acceleration due to gravity on Earth?&amp;quot;,
 &amp;quot;options&amp;quot;: [&amp;quot;...&amp;quot;, &amp;quot;...&amp;quot;, &amp;quot;...&amp;quot;, &amp;quot;...&amp;quot;],
 &amp;quot;answer&amp;quot;: 0,
 &amp;quot;solution&amp;quot;: &amp;quot;Very obvious.&amp;quot;
 }
 ]
 }
 ]
 }
 ]
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;There are three question types: &lt;code&gt;SMCQ&lt;/code&gt; (single correct), &lt;code&gt;MMCQ&lt;/code&gt; (multiple correct, so &lt;code&gt;answer&lt;/code&gt; becomes an array like &lt;code&gt;[0, 2]&lt;/code&gt;) and &lt;code&gt;NUM&lt;/code&gt; (numerical, where &lt;code&gt;answer&lt;/code&gt; is just a number like &lt;code&gt;1.414&lt;/code&gt;). The &lt;code&gt;duration&lt;/code&gt; is in seconds. Images are not embedded - &lt;code&gt;image&lt;/code&gt; and &lt;code&gt;simage&lt;/code&gt; (the solution&amp;rsquo;s image) are plain links to wherever the image happens to be hosted. I am aware this is cheating.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;2. The repo&amp;rsquo;s &lt;code&gt;index.json&lt;/code&gt;.&lt;/strong&gt; Every test repo has one at its root. It lists the tests in that repo, along with just enough metadata for the browser to draw a card without downloading the entire test:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-json"&gt;[
 {
 &amp;quot;name&amp;quot;: &amp;quot;Sample Test&amp;quot;,
 &amp;quot;filename&amp;quot;: &amp;quot;test1.json&amp;quot;,
 &amp;quot;author&amp;quot;: &amp;quot;Sayak&amp;quot;,
 &amp;quot;duration&amp;quot;: 3600,
 &amp;quot;tags&amp;quot;: [&amp;quot;physics&amp;quot;, &amp;quot;mathematics&amp;quot;, &amp;quot;maths&amp;quot;, &amp;quot;jee&amp;quot;],
 &amp;quot;description&amp;quot;: &amp;quot;A sample Physics, Maths test&amp;quot;,
 &amp;quot;url&amp;quot;: &amp;quot;https://raw.githubusercontent.com/librexam/testRepo/main/tests/test1.json&amp;quot;
 }
]
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;3. The central &lt;code&gt;indexedRepos.json&lt;/code&gt;.&lt;/strong&gt; This one lives in my own repo, &lt;a href="https://github.com/sayakdattagupta/librexamtests"&gt;librexamtests&lt;/a&gt;, and is simply a list of every repo found so far - its name, when it was last pushed to, and its star count.&lt;/p&gt;
&lt;h2 id="heres-what-happens"&gt;Here’s what happens&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;The end-user creates an exam (which, obviously, follows a strict format). The file extension is &lt;code&gt;.json&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;They then create a GitHub repository&lt;br&gt;
&lt;strong&gt;OR&lt;/strong&gt;&lt;br&gt;
commit to an already existing repository. Each repository can host one or multiple tests.&lt;/li&gt;
&lt;li&gt;Each repository has a root-level indexing JSON file that defines metadata for these exams. Updating this index is manual and is the responsibility of the repo contributors, for now — though I might help automate this in the future using a GitHub workflow.&lt;/li&gt;
&lt;li&gt;Each repository is tagged with the &lt;code&gt;librexam-test&lt;/code&gt; topic.&lt;/li&gt;
&lt;li&gt;I, on my behalf, have yet another central repository with a workflow that uses the GitHub API to find repos with this tag and index them. Since I am not using an API key here, I am rate-limited.&lt;/li&gt;
&lt;li&gt;My app, LibrExam, simply moves top-down beginning with my repo-that-has-indexed-repos.&lt;/li&gt;
&lt;li&gt;Voila.&lt;/li&gt;
&lt;/ol&gt;
&lt;figure&gt;&lt;img src="https://sayakdattagupta.com/assets/librexambrowse.jpg" alt="librexambrowser" width="1920" height="946" loading="lazy" decoding="async"&gt;&lt;/figure&gt;&lt;p class="caption"&gt;An experience polished enough to attract 10 million dollars in VC funding.&lt;/p&gt;
&lt;h2 id="what-top-down-actually-means"&gt;What “top-down” actually means&lt;/h2&gt;
&lt;p&gt;Step 6 is doing a lot of heavy lifting, so here it is in slightly more detail. When you open the browser, LibrExam:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Fetches &lt;code&gt;indexedRepos.json&lt;/code&gt; from my repo.&lt;/li&gt;
&lt;li&gt;Fetches the &lt;code&gt;index.json&lt;/code&gt; of &lt;em&gt;every single repo&lt;/em&gt; in that list, all at once.&lt;/li&gt;
&lt;li&gt;Merges all of them into one list of test cards, which you can then search, sort and filter by tag.&lt;/li&gt;
&lt;li&gt;Only downloads the actual test when you click on one.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Every response also gets stashed in &lt;code&gt;localStorage&lt;/code&gt;. If GitHub is down, or your Wi-Fi is having a moment, LibrExam quietly falls back to the last copy it saw. Stripped of all the logging, the core of it is this:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-js"&gt;async function fetchIndex(url, cacheKey) {
 try {
 const res = await fetch(`${url}?t=${Date.now()}`);
 if (!res.ok) throw new Error(&amp;quot;Failed to load index&amp;quot;);
 const data = await res.json();
 localStorage.setItem(cacheKey, JSON.stringify(data));
 return data;
 } catch (err) {
 const cached = localStorage.getItem(cacheKey);
 if (cached) return JSON.parse(cached);
 throw err;
 }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That &lt;code&gt;?t=${Date.now()}&lt;/code&gt; is important. It also brings me to the problems.&lt;/p&gt;
&lt;h2 id="there-are-some-problems-with-this-approach"&gt;There are some problems with this approach&lt;/h2&gt;
&lt;p&gt;To begin with, I make a hard-write (by putting a timestamp) on my final indexed repo to avoid missing out on rapidly committed changes from the user-end repositories, regardless of whether there are any changes, thus bypassing cache issues.&lt;/p&gt;
&lt;p&gt;The cache in question is GitHub’s own. Files on &lt;code&gt;raw.githubusercontent.com&lt;/code&gt; are served with &lt;code&gt;Cache-Control: max-age=300&lt;/code&gt; - meaning the copy you receive can be up to five minutes old, no matter how recently it was pushed. The &lt;code&gt;?t=&lt;/code&gt; in the browser exists for the same reason: it makes every single request look like a brand new URL.&lt;/p&gt;
&lt;p&gt;And with that, we come across the next problem. Updates happen only around once every 10 minutes, indiscriminately. There’s no synchronicity in this system. Somewhere, someday in the near future, someone might have to wait up to 10 minutes to try out the Star Wars Trivia 100-marker that their friend has made. Tragically, then, Buzzfeed might eat into LibrExam’s market share.&lt;/p&gt;
&lt;p&gt;Then there is the rate limit. Without a token, the GitHub API hands you 60 requests an hour, and its search endpoint - the one that actually finds &lt;code&gt;librexam-test&lt;/code&gt; repos - a measly 10 a minute, at 100 results a page. That is plenty for two repos. It is decidedly not plenty for two thousand.&lt;/p&gt;
&lt;p&gt;Speaking of two thousand: step 2 of “top-down” means that every visitor’s browser fetches every repo’s &lt;code&gt;index.json&lt;/code&gt;. With two repos, that is three requests. With five hundred, it is five hundred and one - per visitor, per visit. My serverless architecture would then very efficiently DDoS GitHub on my behalf.&lt;/p&gt;
&lt;p&gt;And, my personal favourite: the browser builds the address of each &lt;code&gt;index.json&lt;/code&gt; as &lt;code&gt;raw.githubusercontent.com/&amp;lt;repo&amp;gt;/main/index.json&lt;/code&gt;. The branch name is hardcoded. If your repo’s default branch is still called &lt;code&gt;master&lt;/code&gt; - which, until late 2020, was GitHub’s default for every new repo - LibrExam will simply never see your tests.&lt;/p&gt;
&lt;p&gt;Finally, there is the concern of unmoderated content — a concern expressed by one of my patrons. Truthfully, I must say, I would be very happy if such a concern materialised, because only then could I truly say that my project has been a success.&lt;/p&gt;</description></item><item><title>LibrExam (the birth of)</title><link>https://sayakdattagupta.com/post/2025/07/17/librexam-the-birth-of/</link><pubDate>Thu, 17 Jul 2025 00:00:00 +0000</pubDate><guid>https://sayakdattagupta.com/post/2025/07/17/librexam-the-birth-of/</guid><description>&lt;p&gt;Mock tests are a global phenomenon. Every student — from American AP Physics 2 bros to Chinese Gaokao tryhards to Indian JEE Advanced sweats — takes them. And they usually cost a pretty penny.&lt;/p&gt;
&lt;p&gt;Not anymore.&lt;/p&gt;
&lt;p&gt;Goodbye legacy, proprietary, limited-in-scope-and-content test taking software. Welcome &lt;a href="https://github.com/sayakdattagupta/librexam"&gt;LibrExam&lt;/a&gt;. Welcome its AGPL-3 License (I am looking at you, Microsoft*).&lt;/p&gt;
&lt;p&gt;The fundamental idea behind LibrExam is that it&amp;rsquo;s kind of decentralised. Obviously, using GitHub as a content-sharing-system, though novel, makes it centralised - but users always have the ability to share the JSON (yes, tests are written out in JSONs) file of their test; Moving away from third-party or self-hosted images is a different issue, however - I am still yet to figure that out (LibrExam is very, VERY much in alpha)&lt;/p&gt;
&lt;p&gt;Now, to be absolutely honest, I had toyed with this LibrExam idea for quite a while before finally starting to work on it—mostly because competitive examinations had been taking the lion&amp;rsquo;s share of my blood pressure (bad), the preparation for which had made me use several proprietary quiz platforms (torture).&lt;/p&gt;
&lt;p&gt;Safe to say: All of those platforms are coming down.&lt;/p&gt;
&lt;p&gt;*P.S.: Dear Microsoft - please let me know if you want to buy me out for a 1 million dollar fee.&lt;/p&gt;</description></item><item><title>Subscribe to some RSS Feeds</title><link>https://sayakdattagupta.com/post/2025/05/11/subscribe-to-some-rss-feeds/</link><pubDate>Sun, 11 May 2025 00:00:00 +0000</pubDate><guid>https://sayakdattagupta.com/post/2025/05/11/subscribe-to-some-rss-feeds/</guid><description>&lt;p&gt;RSS (&amp;lsquo;Really Simple Syndication&amp;rsquo;) feeds are an excellent way to stay up-to-date with recent posts on a website that supports them. Basically, think of it as a newsletter or notifier, but without the bloat, and with the added advantage of (usually) being able to read the said article in its entirety.&lt;/p&gt;
&lt;p&gt;To get started with &amp;ldquo;subscribing&amp;rdquo; to RSS feeds, you simply need an RSS reader. I recommend &lt;a href="https://github.com/spacecowboy/Feeder"&gt;Feeder&lt;/a&gt; (I would prefer you use F-Droid, or Droidify, to install it, instead of the Play Store) on Android.&lt;/p&gt;
&lt;p&gt;Open the app, and simply add the feed link. It&amp;rsquo;s &lt;code&gt;https://sayakdattagupta.com/index.xml&lt;/code&gt; for my site. Most sites (including mine) which have a subscription button/rss icon directly link you to the &lt;code&gt;.xml&lt;/code&gt; file, since browsers can read these files. Don&amp;rsquo;t get confused and simply copy this link into the reader.&lt;/p&gt;
&lt;p&gt;Some readers can even find the feed on their own, if you just paste in the site&amp;rsquo;s address - provided the site announces its feed with a &lt;code&gt;&amp;lt;link rel=&amp;quot;alternate&amp;quot; type=&amp;quot;application/rss+xml&amp;quot;&amp;gt;&lt;/code&gt; tag in its &lt;code&gt;&amp;lt;head&amp;gt;&lt;/code&gt;. Mine doesn&amp;rsquo;t (yet), so stick to the link above.&lt;/p&gt;
&lt;p&gt;The beauty about this system is you aren&amp;rsquo;t connected to some bloated, spyware-esque annoying mailing list. It is 100% local and pulls in content whenever you want to. It even supports images!&lt;/p&gt;
&lt;h2 id="what-is-actually-in-the-feed"&gt;What is actually in the feed&lt;/h2&gt;
&lt;p&gt;An RSS feed is just an XML file. That&amp;rsquo;s it. No accounts, no server-side code - Hugo generates mine every time the site is built. Here&amp;rsquo;s (a trimmed version of) the entry for this very post:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-xml"&gt;&amp;lt;rss version=&amp;quot;2.0&amp;quot;&amp;gt;
 &amp;lt;channel&amp;gt;
 &amp;lt;title&amp;gt;Home on Sayak Dattagupta&amp;lt;/title&amp;gt;
 &amp;lt;link&amp;gt;https://sayakdattagupta.com/&amp;lt;/link&amp;gt;
 &amp;lt;item&amp;gt;
 &amp;lt;title&amp;gt;Subscribe to some RSS Feeds&amp;lt;/title&amp;gt;
 &amp;lt;link&amp;gt;https://sayakdattagupta.com/post/2025/05/11/subscribe-to-some-rss-feeds/&amp;lt;/link&amp;gt;
 &amp;lt;pubDate&amp;gt;Sun, 11 May 2025 00:00:00 +0000&amp;lt;/pubDate&amp;gt;
 &amp;lt;guid&amp;gt;https://sayakdattagupta.com/post/2025/05/11/subscribe-to-some-rss-feeds/&amp;lt;/guid&amp;gt;
 &amp;lt;description&amp;gt;&amp;amp;lt;p&amp;amp;gt;RSS (&amp;amp;amp;lsquo;Really Simple Syndication&amp;amp;amp;rsquo;) feeds are...&amp;lt;/description&amp;gt;
 &amp;lt;/item&amp;gt;
 &amp;lt;/channel&amp;gt;
&amp;lt;/rss&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;&amp;lt;channel&amp;gt;&lt;/code&gt; is the site; every &lt;code&gt;&amp;lt;item&amp;gt;&lt;/code&gt; is a post. The &lt;code&gt;&amp;lt;guid&amp;gt;&lt;/code&gt; is what your reader uses to tell posts apart - it is how it knows what is new, and what you have already read. The &lt;code&gt;&amp;lt;description&amp;gt;&lt;/code&gt; carries the article itself, as (escaped) HTML. Some sites put the entire article in there; mine, for now, only includes the first couple of paragraphs (Hugo&amp;rsquo;s default), so you will have to click through for the rest.&lt;/p&gt;
&lt;p&gt;Only want my tech posts? Every tag has its own feed too - for example, &lt;code&gt;https://sayakdattagupta.com/tags/tech/index.xml&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="how-your-reader-knows-something-is-new"&gt;How your reader knows something is new&lt;/h2&gt;
&lt;p&gt;It doesn&amp;rsquo;t. It just asks, every so often (you get to choose how often).&lt;/p&gt;
&lt;p&gt;Downloading the entire file every time would be wasteful, though, so a well-behaved reader asks politely. Along with the feed, the server (GitHub Pages, in my case) sends an &lt;code&gt;ETag&lt;/code&gt; - a fingerprint of the file. Next time, the reader sends that fingerprint back, and if nothing has changed, the server replies with no file at all:&lt;/p&gt;
&lt;pre&gt;&lt;code class="language-http"&gt;GET /index.xml HTTP/1.1
Host: sayakdattagupta.com
If-None-Match: &amp;quot;&amp;lt;the ETag from last time&amp;gt;&amp;quot;

HTTP/1.1 304 Not Modified
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;(&lt;code&gt;Last-Modified&lt;/code&gt; and &lt;code&gt;If-Modified-Since&lt;/code&gt; do the exact same thing, just with dates instead of fingerprints.)&lt;/p&gt;
&lt;p&gt;Notice what is missing here: a list of subscribers. There isn&amp;rsquo;t one, anywhere. Not even I know who is subscribed to my site - my server only ever sees a file being downloaded. Try getting that from a mailing list.&lt;/p&gt;</description></item></channel></rss>