NFS vs SMB transfer speed on a Linux server

Unknown — Tue, 08 Oct 2024 00:00:00 +0000

I also posted the full data in a git repo at https://github.com/jonfk/nfs-smb-benchmarking-experiment</a>.</p>

NFS vs SMB transfer speed on a Linux server</h1>

I have been setting up my homelab server, migrating it from baremetal to Proxmox, a virtualization solution. The whole process is probably a blog post for another time. But while doing the migration, I got bitten by the improvement bug, where I started wanting to improve various features I had decided on a good enough solution at the time. There are probably several blog posts worth of things to discuss just on what I did, but let's not get side tracked.</p>

One of the things was using SMB vs NFS on Linux. I have a Synology (ahem, actually two) NAS which I use for storing large amounts of data that are not actively used on the system that generates it or the data might need to be shared through the filesystem with multiple devices. Some examples are backups, photos, videos, entertainment media, etc. In this particular case, it was a backup share from my Synology DS218+ to this new VM I was setting up.</p>

So in the past, I had tried to use an NFS mount from my Linux server to the NAS but kept hitting issues with permissions and user mappings that I didn't care to resolve at the time. The setup for SMB was simple enough that I decided to just go with it at the time. But the original reason I wanted to try NFS was that it is supposed to be more resource efficient and theoretically faster than SMB. So I finally decided to barrel through all the issues and got it working. Now with 2 working mounting solutions, I wanted to see if I could benchmark the difference.</p>

So here is what I found out. But first, how I ran the tests and some info about the servers.</p>

Setup</h1>

Server</h2>

A Synology DS218+ NAS with the following specs:</p>

</th> </th></tr></thead>

CPU Model</strong></td> Intel Celeron J3355</td></tr>

CPU Architecture</strong></td> 64-bit</td></tr>

CPU Frequency</strong></td> Dual Core 2.0 burst up to 2.5 GHz</td></tr>

Hardware Transcoding Engine</strong></td> H.264 (AVC), H.265 (HEVC), MPEG-2 and VC-1; maximum resolution: 4K (4096 x 2160); maximum frame rate per second (FPS): 30</td></tr>

System Memory</strong></td> 10 GB DDR3L</td></tr>

Total Effective Storage</strong></td> 15.7TB</td></tr>

Drives</strong></td> 2 x Seagate IronWolf Pro ST18000NT001 18TB 7200 RPM</td></tr>

Network Interface</strong></td>

1GbE</td></tr> </tbody></table>

NFS Config</h3>

</th> </th></tr></thead>

Maximum NFS Protocol</strong></td> NFSv4.1</td></tr>

Read Packet Size</strong></td> 8KB</td></tr>

Write Packet Size</strong></td>

8KB</td></tr> </tbody></table>

SMB Config</h3>

</th> </th></tr></thead>

Maximum SMB protocol</strong></td> SMB3</td></tr>

Minimum SMB protocol</strong></td> SMB1</td></tr>

SMB range</strong></td> SMB1,5MB2,SMB2 and Large MTU,SMB3</td></tr>

Transport encryption mode</strong></td> Client defined</td></tr>

Enable server signing:</strong></td> Disable</td></tr>

Enable Opportunistic Locking</strong></td> Enabled</td></tr>

Enable SMB2 lease</strong></td> Disabled</td></tr>

Enable SMB durable handles (Cross-protocol file locking will be disabled)</strong></td>

Disabled</td></tr> </tbody></table>

</th> </th></tr></thead>

Enable Local Master Browser</strong></td> Disabled</td></tr>

Enable DirSort VFS module</strong></td> Disabled</td></tr>

Veto files</strong></td> Disabled</td></tr>

Allow symbolic links within shared folders</strong></td> Enabled</td></tr>

Allow symbolic links across shared folders</strong></td> Disabled</td></tr>

**Disable multiple connections from the same IP address</td> Disabled</td></tr>

Collect debug logs</strong></td> Disabled</td></tr>

Apply default UNIX permissions</strong></td> Enabled</td></tr>

Do not reserve disk space when creating files</strong></td>

Disabled</td></tr> </tbody></table>

Client</h2>

A little confusing since in this case the Linux server is acting as a client to the NAS.</p>

</th>	</th></tr></thead>
CPU Model</strong></td>	AMD Ryzen™ 7 3750H</td></tr>
CPU Architecture</strong></td>	64-bit</td></tr>
CPU Frequency</strong></td>	4 Cores/8 Threads (4MB Cache , up to 4.0 GHz)</td></tr>
System Memory</strong></td>	16 GB DDR4</td></tr>
Network Interface</strong></td>	2.5GbE</td></tr> </tbody></table> Mount Configs</h3> The section of my cloud-init config that defines the NFS and SMB mounts.</p> #cloud-config </span> </span>mounts</span>: </span> - [ </span>"192.168.50.55:/volume1/backups"</span>, </span>"/mnt/nfs/ds218/backups"</span>, </span>"nfs4"</span>, </span>"rw,noatime,nodiratime,vers=4.1,soft,retrans=2,_netdev,async,rsize=32768,wsize=32768,fsc,cto,acl"</span>, </span>"0"</span>, </span>"0" </span>] </span> - [ </span>"//192.168.50.55/backups"</span>, </span>"/mnt/smb/ds218/backups"</span>, </span>"cifs"</span>, </span>"username=notmyusername,password=notmypassword,vers=3.1.1,uid=1000,_netdev"</span>, </span>"0"</span>, </span>"0" </span>] </span></code></pre> Network</h2> The NAS and server are connected wired through a 2.5GbE switch.</p> Methodology</h1> For more details on the exact script see fs_bench.sh</a>.</p> To test the filesystems speed, I ran the fs_bench.sh script which runs fio with the 5 test types on both mounts:</p> Sequential read</li> Sequential write</li> Random read</li> Random write</li> Mixed random read/write</li> </ul> Results</h1> For the raw results see fio_output.txt</a>. Otherwise here a summary created using the analysis.py</a> script follows in the next section.</p> Performance Comparison: nfs vs smb</h2> ❯ python3 analysis.py </span>Performance Comparison: NFS vs SMB </span>================================== </span> </span>SEQUENTIAL_READ: </span> IOPS: NFS: 110.00 SMB: 112.00 Difference: 1.82% </span> Bandwidth: NFS: 111.00 MiB/s SMB: 112.00 MiB/s Difference: 0.90% </span> Avg Latency: NFS: 36091.93 µs SMB: 35625.10 µs Difference: 1.31% </span> P99 Latency: NFS: 42206.00 µs SMB: 35914.00 µs Difference: 17.52% </span> CPU Usage: </span> User: NFS: 0.03% SMB: 0.04% Difference: 0.01% </span> System: NFS: 0.73% SMB: 0.46% Difference: -0.27% </span> Total: NFS: 0.76% SMB: 0.50% Difference: -0.26% </span> </span>SEQUENTIAL_WRITE: </span> IOPS: NFS: 107.00 SMB: 110.00 Difference: 2.80% </span> Bandwidth: NFS: 108.00 MiB/s SMB: 110.00 MiB/s Difference: 1.85% </span> Avg Latency: NFS: 37040.95 µs SMB: 36137.06 µs Difference: 2.50% </span> P99 Latency: NFS: 57410.00 µs SMB: 60556.00 µs Difference: -5.20% </span> CPU Usage: </span> User: NFS: 0.11% SMB: 0.13% Difference: 0.02% </span> System: NFS: 0.65% SMB: 1.23% Difference: 0.58% </span> Total: NFS: 0.76% SMB: 1.36% Difference: 0.60% </span> </span>RANDOM_READ: </span> IOPS: NFS: 7334.00 SMB: 5513.00 Difference: -24.83% </span> Bandwidth: NFS: 28.70 MiB/s SMB: 21.50 MiB/s Difference: -25.09% </span> Avg Latency: NFS: 541.17 µs SMB: 721.68 µs Difference: -25.01% </span> P99 Latency: NFS: 1319.00 µs SMB: 1352.00 µs Difference: -2.44% </span> CPU Usage: </span> User: NFS: 0.80% SMB: 1.05% Difference: 0.25% </span> System: NFS: 2.96% SMB: 2.56% Difference: -0.40% </span> Total: NFS: 3.76% SMB: 3.61% Difference: -0.15% </span> </span>RANDOM_WRITE: </span> IOPS: NFS: 6618.00 SMB: 7691.00 Difference: 16.21% </span> Bandwidth: NFS: 25.90 MiB/s SMB: 30.00 MiB/s Difference: 15.83% </span> Avg Latency: NFS: 599.69 µs SMB: 514.93 µs Difference: 16.46% </span> P99 Latency: NFS: 1532.00 µs SMB: 2008.00 µs Difference: -23.71% </span> CPU Usage: </span> User: NFS: 0.75% SMB: 1.33% Difference: 0.58% </span> System: NFS: 2.94% SMB: 10.33% Difference: 7.39% </span> Total: NFS: 3.69% SMB: 11.66% Difference: 7.97% </span> </span>MIXED_RANDRW: </span> IOPS: NFS: 3531.00 SMB: 3137.00 Difference: -11.16% </span> Bandwidth: NFS: 13.80 MiB/s SMB: 12.30 MiB/s Difference: -10.87% </span> Avg Latency: NFS: 588.18 µs SMB: 598.77 µs Difference: -1.77% </span> P99 Latency: NFS: 1516.00 µs SMB: 2376.00 µs Difference: -36.20% </span> CPU Usage: </span> User: NFS: 0.79% SMB: 1.17% Difference: 0.38% </span> System: NFS: 3.16% SMB: 5.37% Difference: 2.21% </span> Total: NFS: 3.95% SMB: 6.54% Difference: 2.59% </span> </span>Summary: </span> sequential_read: </span> Performance: SMB is faster by 1.82% </span> CPU Efficiency: SMB uses less CPU by 0.26% </span> sequential_write: </span> Performance: SMB is faster by 2.80% </span> CPU Efficiency: NFS uses less CPU by 0.60% </span> random_read: </span> Performance: NFS is faster by 33.03% </span> CPU Efficiency: SMB uses less CPU by 0.15% </span> random_write: </span> Performance: SMB is faster by 16.21% </span> CPU Efficiency: NFS uses less CPU by 7.97% </span> mixed_randrw: </span> Performance: NFS is faster by 12.56% </span> CPU Efficiency: NFS uses less CPU by 2.59% </span></code></pre> Conclusion</h1> The difference ended up being somewhat negligible for sequential reads and writes. With both NFS and SMB getting pretty close to the theoretical limit of 125MB/s of the 1GbE link.</p> The difference in random read was fairly large of about 30% in favor of NFS. While the random write speed was in favor of SMB with it being about 15% faster. And in a mixed workload of random reads and writes, NFS wins with the lightest of margins of about 12%. In these, it does seem like NFS is slightly more CPU efficient but not really by much.</p> Was all this worth it? Not really but at least now I know that id doesn't make much of a difference at least with my current setup. I have been planning on upgrading the NAS and network to 2.5GbE, I wonder if that would make a difference. At least now that I had an easy quick setup to test this, I can do it again once I upgrade.</p> My Personal Self-Hosted Infrastructure Part 1 Unknown — Mon, 09 Sep 2024 00:00:00 +0000 Introduction</h2> Over the past decade, there's been a noticeable shift in how software is delivered and consumed. Many applications that were once standalone programs are now served over the web. This trend towards web-based services and software-as-a-service (SaaS) models has its merits, but it also opens up opportunities for individuals to take control of their digital lives through self-hosting. In this post, I'll explore the reasons for this shift, the benefits of self-hosting, and share my personal journey into building a self-hosted infrastructure.</p> Why Web Services?</h2> The move towards web-based services has several advantages:</p> Personal Benefits:</p> Access to apps and data from anywhere</li> Easier collaboration on shared projects</li> Simple sharing of content like photo galleries or works-in-progress</li> </ul> Technical Benefits:</p> Access to software through APIs</li> Ability to schedule tasks remotely</li> Offloading processing to potentially more powerful servers</li> </ul> While these benefits could theoretically be achieved through desktop applications or command-line interfaces, web services with HTTP JSON APIs provide one of the simplest ways to reap these advantages.</p> The Case for Self-Hosting</h2> Despite the convenience of services like Google Workspace or Dropbox, there are compelling reasons to consider self-hosting:</p> Data Control: You maintain ownership and control over your data.</li> Customization: You can modify and adapt services to your specific needs.</li> Privacy: Your data remains on your own infrastructure.</li> Learning Opportunity: Self-hosting provides hands-on experience with various technologies.</li> </ol> Self-hosting does come with its challenges, but starting small and focusing on useful, frequently-used services can help you build a sustainable personal infrastructure.</p> How It Started: My Self-Hosting Journey</h2> My venture into self-hosting began in November 2017 with the purchase of a Synology DS218+ NAS, equipped with an 8TB WD Red Pro drive. Initially intended for expanded storage and photo backups, I soon discovered its potential as a home server through Docker support.</p> The first services I installed were:</p> Plex: For media organization and playback</li> Transmission: A torrent client</li> </ol> These services, which I had previously run locally, now benefited from operating on a dedicated, always-on server.</p> I started by using the Docker UI provided by Synology, which was sufficient for managing these two services. For remote access, I utilized Synology's DDNS service.</p> As I expanded my setup, I learned valuable lessons about storage management and hardware selection:</p> Understanding RAID: When my original 8TB drive began to fill up, I realized I needed to choose between additional storage (RAID 0) or data redundancy (RAID 1). I opted for RAID 1 to protect my important photos and videos.</li> Scaling Up: To increase storage capacity, I eventually purchased a larger NAS, the Synology DS1019+ with 5 bays. This experience taught me the importance of anticipating future needs when selecting hardware.</li> Start Small, Think Big: If I could advise my past self, I'd recommend thoroughly evaluating needs before investing in hardware. The adage "buy once, cry once" applies well to NAS and server setups. For those unsure about their requirements, starting with a used mini PC can be an excellent way to test use cases before committing to more expensive hardware.</li> </ol> Conclusion</h2> Self-hosting offers a powerful way to take control of your digital life, but it requires careful planning and a willingness to learn. As my journey shows, starting small and gradually expanding can lead to a robust and personalized infrastructure that meets your specific needs.</p> Technical Terms Glossary</h2> RAID (Redundant Array of Independent Disks)</strong>: A storage technology that combines multiple disk drive components into a logical unit for data redundancy and performance improvement. Common RAID levels include: RAID 0: Stripes data across drives for improved performance but no redundancy.</li> RAID 1: Mirrors data on two drives for full redundancy.</li> </ul> </li> DDNS (Dynamic DNS)</strong>: A method of automatically updating a name server in the Domain Name System with the active DHCP address. This is particularly useful for accessing home servers with dynamic IP addresses.</li> NAS (Network Attached Storage)</strong>: A file-level computer data storage server connected to a computer network providing data access to heterogeneous network clients.</li> Docker</strong>: A platform for developing, shipping, and running applications in containers, which are lightweight, standalone, executable packages of software that include everything needed to run an application.</li> </ul> RFC Processes for Software Development Unknown — Wed, 14 Aug 2024 00:00:00 +0000 Documenting at a high level what you want to achieve before committing to the work is often a good idea. Especially for complex projects or ideas. This allows others to look over the proposed changes, poke holes at it or be made aware of changes that could affect them.</p> A few ways I have seen this done at different companies are Technical Designs (TDs), Architecture Decision Records (ADRs) and Request for Comments.</p> I have found that a great way to write these kind of documents is in a format not too dissimilar to IETF RFCs. Those often have the right level of detail and a structure that lends itself well to software ideas. Words such as must, shall, should, etc as defined in RFC-2119</a> are also useful to communicate the right level of requirement.</p> For documenting decisions at a project level, such as a tool or individual service, ADRs can be useful where a change can affect the overall project or projects interacting with it. I found that ADRs often break down and don't work well when multiple teams and projects are in one Architecture Decision Log (ADL, a collection of ADRs). They work better in high context situations where ADRs can be short and teams have enough understanding of the project to not have too much background details in ADRs.</p> Depending on your company or situation, you might want to make these documents more or less formal and structured.</p> One good example of an RFC process used at a private company for which their documents are public is Oxide Computer with their RFD documents. Requests for Discussion</a>.</p> Trying to understand how Next.js render server components on updates Unknown — Fri, 31 May 2024 00:00:00 +0000 I was reading the Next.js App Router course</a>, mostly for my own front end curiosity but also because I was looking at doing some frontend projects, as one does when you are frontend curious but mostly do backend development. When I got to Chapter 11: Adding Search and Pagination</a>, I was having a hard time really understanding what was happening and how Next was rendering the updates to the page.</p> First let's start with what this part of the Next.js tutorial is about and how it explains it.</p> You start with a fairly static page.</p> import </span>Pagination </span>from </span>'@/app/ui/invoices/pagination'</span>; </span>import </span>Search </span>from </span>'@/app/ui/search'</span>; </span>import </span>Table </span>from </span>'@/app/ui/invoices/table'</span>; </span>import </span>{ CreateInvoice } </span>from </span>'@/app/ui/invoices/buttons'</span>; </span>import </span>{ lusitana } </span>from </span>'@/app/ui/fonts'</span>; </span>import </span>{ InvoicesTableSkeleton } </span>from </span>'@/app/ui/skeletons'</span>; </span>import </span>{ Suspense } </span>from </span>'react'</span>; </span> </span>export default async </span>function </span>Page</span>() { </span> </span>return </span>( </span> <</span>div </span>className</span>=</span>"w-full"</span>> </span> <</span>div </span>className</span>=</span>"flex w-full items-center justify-between"</span>> </span> <</span>h1 </span>className</span>=</span>{</span>`${lusitana.className} text-2xl`</span>}>Invoices</</span>h1</span>> </span> </</span>div</span>> </span> <</span>div </span>className</span>=</span>"mt-4 flex items-center justify-between gap-2 md:mt-8"</span>> </span> <</span>Search </span>placeholder</span>=</span>"Search invoices..." </span>/> </span> <</span>CreateInvoice </span>/> </span> </</span>div</span>> </span> {</span>/* <Suspense key={query + currentPage} fallback={<InvoicesTableSkeleton />}> </span> <Table query={query} currentPage={currentPage} /> </span> </Suspense> /</span>} </span> <</span>div </span>className</span>=</span>"mt-5 flex w-full justify-center"</span>> </span> {</span>/ <Pagination totalPages={totalPages} /> /</span>} </span> </</span>div</span>> </span> </</span>div</span>> </span> ); </span>} </span></code></pre> The 2 main components we care about are the <Search></code> and <Table></code> components. The <Search></code> component is the input that will take a query and somehow update the results in the <Table></code>. The tutorial explains that the best way to handle this is through using the page's url query params with useSearchParams</code>.</p> The Search</code> component then looks like this:</p> 'use client'</span>; </span> </span>import </span>{ MagnifyingGlassIcon } </span>from </span>'@heroicons/react/24/outline'</span>; </span>import </span>{ useSearchParams, usePathname, useRouter } </span>from </span>'next/navigation'</span>; </span> </span>export default </span>function </span>Search</span>() { </span> </span>const </span>searchParams </span>= </span>useSearchParams</span>(); </span> </span>const </span>pathname </span>= </span>usePathname</span>(); </span> </span>const </span>{ replace } </span>= </span>useRouter</span>(); </span> </span> </span>function </span>handleSearch</span>(</span>term</span>: </span>string</span>) { </span> </span>const </span>params </span>= new </span>URLSearchParams(searchParams); </span> </span>if </span>(term) { </span> params.</span>set</span>(</span>'query'</span>, term); </span> } </span>else </span>{ </span> params.</span>delete</span>(</span>'query'</span>); </span> } </span> </span>replace</span>(</span>`${pathname}?${params.</span>toString</span>()}`</span>); </span> } </span>//... </span>} </span></code></pre> Note that part of the component was elided. See the full component in the Git repo</a>.</p> Next, we need to wire the query params that were updated by our Search</code> component into the <Table></code> component.</p> The page will now look like this:</p> import </span>Pagination </span>from </span>'@/app/ui/invoices/pagination'</span>; </span>import </span>Search </span>from </span>'@/app/ui/search'</span>; </span>import </span>Table </span>from </span>'@/app/ui/invoices/table'</span>; </span>import </span>{ CreateInvoice } </span>from </span>'@/app/ui/invoices/buttons'</span>; </span>import </span>{ lusitana } </span>from </span>'@/app/ui/fonts'</span>; </span>import </span>{ Suspense } </span>from </span>'react'</span>; </span>import </span>{ InvoicesTableSkeleton } </span>from </span>'@/app/ui/skeletons'</span>; </span> </span>export default async </span>function </span>Page</span>({ </span> </span>searchParams</span>, </span>}</span>: </span>{ </span> searchParams</span>?: </span>{ </span> query</span>?: </span>string</span>; </span> page</span>?: </span>string</span>; </span> }; </span>}) { </span> </span>const </span>query </span>= </span>searchParams?.query </span>\|\| </span>''</span>; </span> </span>const </span>currentPage </span>= </span>Number</span>(searchParams?.page) </span>\|\| </span>1</span>; </span> </span> </span>return </span>( </span> <</span>div </span>className</span>=</span>"w-full"</span>> </span> <</span>div </span>className</span>=</span>"flex w-full items-center justify-between"</span>> </span> <</span>h1 </span>className</span>=</span>{</span>`${lusitana.className} text-2xl`</span>}>Invoices</</span>h1</span>> </span> </</span>div</span>> </span> <</span>div </span>className</span>=</span>"mt-4 flex items-center justify-between gap-2 md:mt-8"</span>> </span> <</span>Search </span>placeholder</span>=</span>"Search invoices..." </span>/> </span> <</span>CreateInvoice </span>/> </span> </</span>div</span>> </span> <</span>Suspense </span>key</span>=</span>{query </span>+ </span>currentPage} </span>fallback</span>=</span>{<</span>InvoicesTableSkeleton </span>/>}> </span> <</span>Table </span>query</span>=</span>{query} </span>currentPage</span>=</span>{currentPage} /> </span> </</span>Suspense</span>> </span> <</span>div </span>className</span>=</span>"mt-5 flex w-full justify-center"</span>> </span> {</span>/ <Pagination totalPages={totalPages} /> /</span>} </span> </</span>div</span>> </span> </</span>div</span>> </span> ); </span>} </span></code></pre> Now, whenever the <Search></code> component updates the query params with the search query entered, the <Table></code> will have the search query passed into it's props so that it can show the right results.</p> Finally, let's take a look at what's happening in the <Table></code> component.</p> // ... </span>export default async </span>function </span>InvoicesTable</span>({ </span> </span>query</span>, </span> </span>currentPage</span>, </span>}</span>: </span>{ </span> query</span>: </span>string</span>; </span> currentPage</span>: </span>number</span>; </span>}) { </span> </span>const </span>invoices </span>= await </span>fetchFilteredInvoices</span>(query, currentPage); </span> </span>// ... </span>} </span></code></pre> This shows that the <Table></code> component calls an async function fetchFilteredInvoices</code> with the query and currentPage obtained from the Props of the component. This async function could actually fetch the invoices from any data source. In the Next.js tutorial, it's from a database covered in a previous chapter.</p> The tutorial for this chapter actually continues and covers a few more things: input debouncing (which which delays triggering any change until what is triggering the change stops, in this case it's the user typing) and pagination. But for our purposes, we can stop here and summarize what we and the tutorial covers and see we can understand what happened.</p> Quick Summary</h1> Our page has 2 components, a <Search></code> and a <Table></code> component. These 2 components interact with each other through the url query parameters of the page.</p> The <Search></code> component takes user input of the search query and updates the query parameter of the page with that search query. e.g. the user enters Lee</code> the url changes to /dashboard/invoices?query=Lee</code>.</p> The <Table></code> component takes the query parameters as it's Props and renders the right data.</p> When the user types a new query into the search box, the <Table></code> component updates the data.</p> But how is this all happening? Does the page refresh when the query parameters are updated? How does the table receive the updated data based on the search terms?</p> Maybe I am particularly thick or I might have missed something while reading this tutorial sleepy right before bed. But I had all these questions popping up in my head while reading this Chapter.</p> Chapter 9: Streaming</a> does discuss how you can chunks of a page by loading partial content initially and then streaming in the rest of the content is chunks. This uses React <Suspense></code> feature or the Next.js loading.tsx</code> feature. Maybe Next.js is streaming in the data to the component?</p> Chapter 10: Partial Prerendering</a> covers how a page can be both static and dynamic. It describes the feature at a high level but I am not entirely sure if this is related to what I am missing.</p> Let's see how it works</h1> I have often found that the best way to understand how something works is to build and poke at what you want to understand. In this case, since this is a web thing, we can easily poke at it using dev tools. So let's do it.</p> We can reuse most of the code from the tutorial, except for the fetchFilteredInvoices</code> function which uses a database and was setup in Chapter 6 and 7</a>. Instead we will mock this and use the data provided in the tutorial to populate the database.</p> import </span>{ customers, invoices } </span>from </span>'@/app/lib/placeholder-data'</span>; </span> </span>const </span>FETCH_SIMULATED_DELAY </span>= </span>3000</span>; </span>// delay in milliseconds </span> </span>export async </span>function </span>fetchFilteredInvoices</span>(</span>query</span>: </span>string</span>, </span>currentPage</span>: </span>number</span>)</span>: </span>Promise<{ </span> name</span>: </span>string </span>\| </span>undefined</span>, </span> email</span>: </span>string </span>\| </span>undefined</span>, </span> image_url</span>: </span>string </span>\| </span>undefined</span>, </span> customer_id</span>: </span>string</span>, </span> amount</span>: </span>number</span>, </span> status</span>: </span>string</span>, </span> date</span>: </span>string </span>}[]> { </span> </span>return new </span>Promise((</span>resolve</span>) </span>=> </span>{ </span> </span>setTimeout</span>(() </span>=> </span>{ </span> </span>// Combine invoices with customer details </span> </span>const </span>detailedInvoices </span>= </span>invoices.</span>map</span>(</span>invoice </span>=> </span>{ </span> </span>const </span>customer </span>= </span>customers.</span>find</span>(</span>c </span>=> </span>c.id </span>=== </span>invoice.customer_id); </span> </span>return </span>{ </span> </span>...</span>invoice, </span> name: customer?.name, </span> email: customer?.email, </span> image_url: customer?.image_url, </span> }; </span> }); </span> </span> </span>// Simulate filtering based on the query </span> </span>const </span>filteredInvoices </span>= </span>detailedInvoices.</span>filter</span>(</span>invoice </span>=> </span> invoice.name?.</span>toLowerCase</span>().</span>includes</span>(query.</span>toLowerCase</span>()) </span> ); </span> </span> </span>// Simulate pagination </span> </span>const </span>pageSize </span>= </span>10</span>; </span> </span>const </span>startIndex </span>= </span>(currentPage </span>- </span>1</span>) </span> </span>pageSize; </span> </span>const </span>paginatedInvoices </span>= </span>filteredInvoices.</span>slice</span>(startIndex, startIndex </span>+ </span>pageSize); </span> </span> </span>resolve</span>(paginatedInvoices); </span> }, FETCH_SIMULATED_DELAY); </span>// Simulate an x second delay </span> }); </span>} </span></code></pre> For good measure, I also added a delay to the fetchFilteredInvoices</code> function to simulate calling an external data source.</p> Let's run this example and see what's happening.</p> ></span> npm run dev </span></code></pre> ![first_page_load] Insert screenshot here</p> On first page load everything is as expected. The initial request is a call to /dashboard/invoices</code> which gets an html response. This initial request is followed by several various js requests to load the script to bootstrap the client side application.</p> But what happens when we enter a search query?</p> ![search_query_partial_render] Insert screenshot</p> We can see that the application does 2 requests. We can finally see what they are doing by inspecting them. What we see are that the requests on /dashboard/invoices</code> with specific headers and query parameters, return a react tree response. This react tree is the part of the page that were updated. The first request is the initial render of the Skeleton Table in the <Suspense></code> tags while the fetchFilteredInvoices</code> loaded the data to be rendered. The second request is then the <Table></code> component rendered with the loaded data.</p> // 1. First Request </span>await </span>fetch</span>(</span>"http://localhost:3000/dashboard/invoices?query=Lee&_rsc=14zw6"</span>, { </span> </span>"credentials"</span>: </span>"include"</span>, </span> </span>"headers"</span>: { </span> </span>"User-Agent"</span>: </span>"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"</span>, </span> </span>"Accept"</span>: </span>"/"</span>, </span> </span>"Accept-Language"</span>: </span>"en-US,en;q=0.5"</span>, </span> </span>"RSC"</span>: </span>"1"</span>, </span> </span>"Next-Router-State-Tree"</span>: </span>"%5B%22%22%2C%7B%22children%22%3A%5B%22dashboard%22%2C%7B%22children%22%3A%5B%22invoices%22%2C%7B%22children%22%3A%5B%22__PAGE__%22%2C%7B%7D%5D%7D%5D%7D%5D%7D%2Cnull%2Cnull%2Ctrue%5D"</span>, </span> </span>"Next-Router-Prefetch"</span>: </span>"1"</span>, </span> </span>"Next-Url"</span>: </span>"/dashboard/invoices"</span>, </span> </span>"Sec-Fetch-Dest"</span>: </span>"empty"</span>, </span> </span>"Sec-Fetch-Mode"</span>: </span>"cors"</span>, </span> </span>"Sec-Fetch-Site"</span>: </span>"same-origin"</span>, </span> </span>"Priority"</span>: </span>"u=4" </span> }, </span> </span>"referrer"</span>: </span>"http://localhost:3000/dashboard/invoices"</span>, </span> </span>"method"</span>: </span>"GET"</span>, </span> </span>"mode"</span>: </span>"cors" </span>}); </span> </span>// 2. Second Request </span>await </span>fetch</span>(</span>"http://localhost:3000/dashboard/invoices?query=Lee&_rsc=181do"</span>, { </span> </span>"credentials"</span>: </span>"include"</span>, </span> </span>"headers"</span>: { </span> </span>"User-Agent"</span>: </span>"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"</span>, </span> </span>"Accept"</span>: </span>"/"</span>, </span> </span>"Accept-Language"</span>: </span>"en-US,en;q=0.5"</span>, </span> </span>"RSC"</span>: </span>"1"</span>, </span> </span>"Next-Router-State-Tree"</span>: </span>"%5B%22%22%2C%7B%22children%22%3A%5B%22dashboard%22%2C%7B%22children%22%3A%5B%22invoices%22%2C%7B%22children%22%3A%5B%22__PAGE__%3F%7B%5C%22query%5C%22%3A%5C%22Lee%5C%22%7D%22%2C%7B%7D%2Cnull%2C%22refetch%22%5D%7D%5D%7D%5D%7D%5D"</span>, </span> </span>"Next-Url"</span>: </span>"/dashboard/invoices"</span>, </span> </span>"Sec-Fetch-Dest"</span>: </span>"empty"</span>, </span> </span>"Sec-Fetch-Mode"</span>: </span>"cors"</span>, </span> </span>"Sec-Fetch-Site"</span>: </span>"same-origin"</span>, </span> </span>"Priority"</span>: </span>"u=4" </span> }, </span> </span>"referrer"</span>: </span>"http://localhost:3000/dashboard/invoices"</span>, </span> </span>"method"</span>: </span>"GET"</span>, </span> </span>"mode"</span>: </span>"cors" </span>}); </span></code></pre> 0:["development",[["children","dashboard","children","invoices","children","__PAGE__?{\"query\":\"Lee\"}",["__PAGE__?{\"query\":\"Lee\"}",{}],null,null]]] </span></code></pre> 0:["development",[["children","dashboard","children","invoices","children","__PAGE__?{\"query\":\"Lee\"}",["__PAGE__?{\"query\":\"Lee\"}",{}],"$L1",[null,"$L2"]]]] </span>1:["$L3","$L4",null] </span>2:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","meta","2",{"name":"next-size-adjust"}]] </span>5:I["(app-pages-browser)/./app/ui/search.tsx",["app/dashboard/invoices/page","static/chunks/app/dashboard/invoices/page.js"],""] </span>6:"$Sreact.suspense" </span>4:["$","div",null,{"className":"w-full","children":[["$","div",null,{"className":"flex w-full items-center justify-between","children":["$","h1",null,{"className":"__className_712214 text-2xl","children":"Invoices"}]}],["$","div",null,{"className":"mt-4 flex items-center justify-between gap-2 md:mt-8","children":["$","$L5",null,{"placeholder":"Search invoices..."}]}],["$","$6","Lee1",{"fallback":["$","div",null,{"className":"mt-6 flow-root","children":["$","div",null,{"className":"inline-block min-w-full align-middle","children":["$","div",null,{"className":"rounded-lg bg-gray-50 p-2 md:pt-0","children":[["$","div",null,{"className":"md:hidden","children":[["$","div",null,{"className":"mb-2 w-full rounded-md bg-white p-4","children":[["$","div",null,{"className":"flex items-center justify-between border-b border-gray-100 pb-8","children":[["$","div",null,{"className":"flex items- </span>... </span></code></pre> Summary</h1> So we were able to answer some of the questions we had initially. Next.js manages the request lifecycle of the application. It does a full page load initially where it is necessary, but once loaded, replaces only parts of the page that are necessary. No full page reload happens when the query params are updated on the page or when new data is rendered to the page. The data is passed through the page's url query parameters mediated through next/navigation</code> which manages the updates to the page when necessary.</p> sequenceDiagram </span> User ->> UserAgent: Load /dashboard/invoices </span> UserAgent ->> Next.js Server: Request /dashboard/invoices </span> activate Next.js Server </span> Next.js Server -->> UserAgent: text/html </span> deactivate Next.js Server </span> </span> User ->> UserAgent: Enters search query </span> UserAgent ->> Next.js Server: Request /dashboard/invoices?query=...&_rsc=... </span> activate Next.js Server </span> Note right of Next.js Server: Next.js renders the changes </span> Next.js Server -->> UserAgent: text/x-component of the changes </span> deactivate Next.js Server </span></code></pre> Check out the repo where I tried to create a minimal example of what I looked into here. Github link</a></p> Playing with Automerge and text CRDTs Unknown — Sat, 01 Feb 2020 00:00:00 +0000 You will find the code and output from the work described here at the following links:</p> Github Repo</a></li> Web App</a></li> </ul> Automerge: a Javascript CRDT library</h2> I had heard of Conflict-free replicated data types</a> before but their use crystalized in my mind recently when I found out about the Automerge</a> Javascript library. It's blurb states that it's a JSON-like data structure that can be modified concurrently and merged automatically. What this means is that it enables the creation of collaborative applications where multiple users could modify the same structure locally, send their changes over the network and have their changes merge together even when they made conflicting modifications. It is of course not perfect since the library cannot know in advance what the right state would be if conflicting modifications happened on the same property if it doesn't actually support it, in these cases, it chooses one of the updates and saves the other in it's conflicts. What's great about it's conflict resolution, is that when it chooses one resolution, all the nodes using automerge receiving the conflicting changes will also resolve it in the same consistent way.</p> One of the data structures I was interested on was the Text</code> data type, which promises to enable text editors to become collaborative and allow concurrent updates even offline. With that in mind, a lot of use cases started to pop into my mind, since I have been thinking about a better way of storing my notes recently. It matches several criteria that could help be useful for my use case, namely, being in Javascript, I could use it to build a web app to sync my notes in several places and even enable merging conflicting updates when I come back online from different devices.</p> But I haven't been working much with frontend web technologies recently, so I decided to start small and experiment with the library with a small prototype. This is what I learned while doing that.</p> React and Redux</h2> Any good frontend project has to start with some kind of rendering library or framework. I could go vanilla js, or choose one of the many frontend libraries available today. One that seems to have largely taken over the frontend world these days seem to be React. I used React at a startup several years ago and from a cursory look, the ecosystem seems to have moved forward quite well, the APIs are largely the same, but the breath of libraries, tooling and documentation has improved a lot in the intervening years. So I decided to go with a React/Redux rendering stack.</p> This architecture is quite similar to Elm</a>, which I have been using in some of my side projects. I should probably write about it sometime, but today isn't about Elm. Let's just say that Elm is a functional programming language for frontend development with a strong focus on no runtime errors (at least not ones of the typed kind). Since it's the tech I have been using the most in my side projects where I need a web frontend, that is the starting point for my comparisons.</p> The first thing I noticed digging around is how much more breath of libraries and integrations there are for a React application. By comparison in Elm, I found myself feeling a little isolated by lack of libraries or different tooling from the rest of the web ecosystem.</p> I also found the improved documentation around React since the last time I used it to be very nice. The React, Redux and Create React App official websites had a lot of examples and pragmatic use cases to help.</p> The onboarding process for a new project seems to have been streamlined greatly since I used React. The Create React App and react-scripts enables you to have a working project from a template and configuration less build pretty quick. You can of course configure your own bundler but I enjoy being able to start on my project fairly quick and only needing to configure the bundler once I have a specific need to do that.</p> Typescript</h2> I am not big fan of Javascript, it's dynamic type system, it's prototype base inheritance or it's sometimes confusing scoping rules and it's many other faults don't help when I don't use the language often. So seeing typescript coming up in popularity in the Javascript ecosystem fills me with much joy. Even if Typescript doesn't have the strictest type system or one that covers all cases that can occur in Javascript, it's still better than nothing. That is why I wanted to get some familiarity with it.</p> After about 15mins on its documentation</a>, I found myself having enough of a grasp of the language to start using it in a project. So that's encouraging. React</a>, Redux</a> and create-react-app</a> all have some documentation for using them with typescript, which is also a huge help to get started. I also found that most libraries that I encountered had typescript types either included with the library or as a @type/</code> package.</p> The only issue I encountered was went I tried to use immer</a> for my redux reducers. It cause strange type errors causing the Automerge objects stored in my redux store to not be typed as Automerge types because of the readonly properties added by immer it seemed. I didn't really debug much further and quickly reverted back to plain reducers. If I were to use it in the future, I would try to debug the issue, but that does give me pause about adding libraries and their interactions between each other and typescript.</p> Issues with Webpack prod build</h2> The final problem I encountered was at the very end of my experiment. During my whole development, I had been using it's development mode to run my app and it was working fine until I decided to run npm run build</code>. The production build started to give me undefined errors on a class method I was calling on an Automerge object. Looking strictly at the code was no help since from the typescript source code, I was calling the method correctly to bind this</code> and it was working correctly on the development build. It was only after asking for some help from a coworker with much more experience on the front end that we ended up debugging the issue to the bundled and minified code from webpack. It required disabling source maps and locating the corresponding code in the transpiled output using judiciously placed unique literal strings that we found that the code wasn't being transpiled to the same output we would expect. The solution we ended up implementing was to add a .bind()</code> which looked something like this.</p> src/utils/automerge.js</div> ... </span>doc.text.insertAt?.</span>bind</span>(doc.text)(idx, changedText); </span>... </span></code></pre> This did the trick but doesn't inspire me with a ton of confidence in the tooling in the Javascript ecosystem.</p> Conclusions</h2> This has been a bit of an off the cuff post on my experience getting back into frontend development and my experiment with Typescript and the Automerge CRDT library. Overall, I found that the Javascript ecosystem has a huge breath of libraries to do almost anything. That Typescript is pretty awesome but the integration with the libraries you want to use might not be perfect. Bringing libraries to your project can have lots of risks and you should only bring them in if you feel confident in truly owning those dependencies, and that's especially true in an ecosystem with so many libraries.</p> Finally, the goal of the experiment to see if Automerge was a viable data structure for a generic note editor, indicated to me that although it works amazingly well, it may not be the best match for the potential generic note editor project I had in mind. The updates to the Text data structure took a bit too long (moderate updates to the text sometimes took over 4s) and the storage overhead of the data structure was also way too high (3KB of text took 650KB of storage for the data structure which is about a 200x overhead).</p> Check out the code in the Github repo</a> and the web app</a>.</p> I will keep an eye out for it in the future as it improves. The author is working on improving the performance of the Text data structure and overall library as these issue indicate.</p> https://github.com/automerge/automerge/issues/212</li> https://github.com/automerge/automerge/issues/89</li> </ul> Integrating Ory Hydra OAuth 2.0 with a Rust web service Unknown — Sat, 18 Jan 2020 00:00:00 +0000 I have been using Rust quite a bit more over the last year in a variety of use cases and I have been quite satisfied with the language, libraries and the community. So I think it might be time to write a little bit about how well suited Rust works in one area that I have worked a lot in, and that is HTTP APIs.</p> For this post I will be covering using Rust to write a web service that integrates with the Ory Hydra OAuth 2.0 Server and OpenID Connect Provider</a>. I think that might give a decent idea of the features offered by Rust and Rust libraries for web development.</p> Before we start, all the code we will be working on will be pushed to the following repository</a></p> What will we be doing in this post:</p> What is OAuth 2.0?</a> When should you used it?</a></li> Implementing authorization with OAuth 2.0</a></li> </ul> </li> Hydra: An OpenID Certified OAuth 2.0 Server and OpenID Connect Provider</a> Integrating with Hydra</a></li> </ul> </li> A Rust web service</a> Hello World</a></li> Login and consent</a></li> Setting up the Hydra services</a></li> Generating the Hydra OpenApi Client</a></li> Updating our login and consent implementation</a> Enabling logging</a></li> </ul> </li> Testing our login and consent integration</a></li> Async Rust</a></li> </ul> </li> Where to go from here?</a></li> References</a></li> </ul> What is OAuth 2.0?</h2> The OAuth 2.0 authorization framework enables a third-party application to obtain limited access to an HTTP service, either on behalf of a resource owner by orchestrating an approval interaction between the resource owner and the HTTP service, or by allowing the third-party application to obtain access on its own behalf. From RFC 6749: The OAuth 2.0 Authorization Framework</a></p> </blockquote> Essentially OAuth 2.0 enables applications (servers, mobile apps, SPAs, etc) to obtain access tokens to access an HTTP service scoped only to authorized actions/resources without needing the username and password of the account it would like to obtain access on behalf of.</p> As an almost too short to be correct summary of the OAuth 2.0 Authorization Framework, it defines 2 endpoints, the authorization and the token endpoints. The first is to request authorization with the scopes one would like to receive and the second is to request tokens. It also defines a set of grant types (authorization code, client credentials, etc...) and flows to follow for the different grant types. As for crypto, the protocol simply defines that it needs to be over https. So what we will be doing here won't actually be to spec until we put TLS termination into our system.</p> There are also several extensions to the protocol, the most important one we will be interested in in this post is the Introspection endpoint</a> which defines how resource servers communicate with the authorization server to verify access tokens. Another important one that might be of interested but that we won't cover in this post is the OpenID Connect 1.0 specification</a> which is a protocol build over OAuth 2.0.</p> An example use case where we would use OAuth 2 is when you would like to give access to your youtube account to an application to manage your comments. You wouldn't want that application to also be able to post videos for you or change your profile page. You can do this with OAuth 2.0 by authorizing that application if it has an OAuth 2.0 client with Youtube. That application can go through an OAuth 2.0 flow that will give it an access token that it can then use to get access to manage your comments.</p> Although at first glance, the protocol may sound simple, it is anything but, implementing a spec compliant and interoperable version of the protocol that doesn't have security holes, takes some serious work and no one should trust such an implementation without at least an audit. So how can we use it then if we don't have deep pockets and all the time in the world to write one? That's what we will try to answer in this post using an existing and well tested OAuth 2.0 server with a well documented API.</p> When should you used it?</h3> Given the additional complexity that implementing and integration an OAuth 2.0 server adds to a project, why should you use OAuth 2?</p> The rule of thumb for when you need an OAuth 2.0 implementation, is if you need the ability to authorize third party applications to access your services. Otherwise, if you only need the ability to authorize users on your services through first party clients or applications, an encrypted cookie may better serve you.</p> For more information, check out the following post describing different types of access control and authorization patterns</a>.</p> Implementing authorization with OAuth 2.0</h3> Although it is of course possible to implement an OAuth 2.0 server from scratch, it is strongly not recommended to do so. Getting something wrong with the fundamental building blocks of the security of your system is not to be taken lightly. Another way to implement an OAuth 2 server is with your language's trusted OAuth 2 libraries. Being maintained and kept updated by domain experts, means that your service will have more eyes to find bugs as more people would be using them and gives your an OAuth implementation almost for free.</p> But after having implemented and maintained OAuth2 servers with the Spring OAuth 2 library in Java, I have grown a strong distaste towards this solution too. The reasons for that could be a post in and of itself, but most of the issues stem from maintenance of such such servers when they also contain much of the business logic for the access and authorization rules. It doesn't mean that this needs to be the case but it is often the case in my experience when we go this way.</p> That is why my preference nowadays lean towards the 3rd solution to implementing an authorization server, by using an external blessed implementation with good integration mechanisms. This forces you to keep your business logic and custom authorization rules seperate from a pure spec OAuth 2.0 or OpenID Connect server.</p> Hydra: An OpenID Certified OAuth 2.0 Server and OpenID Connect Provider</h2> This is where Hydra</a> comes in, a hardened, OpenID Certified OAuth 2.0 Server and OpenID Connect Provider optimized for low-latency, high throughput, and low resource consumption. Instead of implementing an OAuth2 server yourself, having to verify your implementation complies with the spec and doesn't have any security holes, we can use Hydra to provide the OAuth2 server functionality and implement the authentication and authorization logic as separate services.</p> Integrating with Hydra</h3> To integrate with the Hydra OAuth2 server, we need to implement a login and consent application. You can review the documentation for doing so here</a>.</p> A Rust web service</h2> If you have never programmed in rust, I would suggest The Rust Book</a> as the introduction to the language.</p> First lets start with a rust web service. Let's create a rust project.</p> cargo new hydra-auth-example-rs </span></code></pre> Now inside of the newly created project directory called hydra-auth-example-rs</code>, you should find 2 things, a src directory with a main.rs</code> file and a Cargo.toml</code> file.</p> $ tree </span>. </span>├── Cargo.toml </span>└── src </span> └── main.rs </span> </span>1 directory, 2 files </span></code></pre> To do anything we will need some dependencies to get our server working. Add the following below in [dependencies]</code>.</p> cargo.toml</div> ...</span> </span>[dependencies] </span>warp </span>= { </span>git </span>= </span>"https://github.com/seanmonstar/warp.git"</span>, </span>rev </span>= </span>"e94309e274872efed34e2a80f1e4553a45963510" </span>} </span>tokio </span>= </span>"0.2.6" </span> </span>log </span>= </span>"0.4.8" </span>env_logger </span>= </span>"0.7" </span></code></pre> Hello World</h3> We can now use these dependencies to provide a basic web service. Let's write a basic hello world endpoint as in the example.</p> src/main.rs</div> use </span>warp::{self, path, Filter}; </span> </span>fn </span>main</span>() { </span> </span>// GET /hello/warp => 200 OK with body "Hello, warp!" </span> </span>let</span> hello </span>= </span>path!(</span>"hello" </span>/ </span>String</span>).</span>map</span>(\|</span>name</span>\| format!(</span>"Hello, </span>{}</span>!"</span>, name)); </span> </span> warp::serve(hello).</span>run</span>(([</span>127</span>, </span>0</span>, </span>0</span>, </span>1</span>], </span>3000</span>)); </span>} </span></code></pre> We can run this hello world service with the following command in the project directory.</p> cargo run </span></code></pre> Or to create a release build.</p> cargo build --release </span># and to run this we will find the built binary at </span>./target/release/hydra-auth-example-rs </span></code></pre> To test this endpoint, you can call it with curl.</p> $ curl http://localhost:3000/hello/jon </span>Hello, jon!% </span></code></pre> Since we will also need login and consent pages for the Hydra integration, let's test html templating. For that we will be using the Tera templating library</a>. There are many other templating libraries in rust and you can find a non-exhaustive list here</a>. So let's add that dependency and use it.</p> Cargo.toml</div> ...</span> </span>[dependencies] </span>...</span> </span> </span>tera </span>= </span>"1.0.1" </span></code></pre> src/main.rs</div> use </span>tera::{Context, Tera}; </span>use </span>warp::{self, path, Filter}; </span> </span>#[tokio::main] </span>async </span>fn </span>main</span>() { </span> </span>let </span>mut</span> tera </span>= </span>Tera::default(); </span> tera.</span>add_raw_template</span>( </span> </span>"hello.html"</span>, </span> </span>r</span>#" </span><!DOCTYPE html> </span><head> </span> <meta charset="UTF-8"> </span> <title>Hello</title> </span></head> </span><body> </span><p>Hello, {{ name }}!</p> </span></body> </span></html> </span>"#</span>, </span> ) </span> .</span>unwrap</span>(); </span> </span> </span>let</span> with_tera </span>= </span>warp::any().</span>map</span>(</span>move \|\|</span> tera.</span>clone</span>()); </span> </span> </span>let</span> hello </span>= </span>path!(</span>"hello" </span>/ </span>String</span>) </span> .</span>and</span>(with_tera) </span> .</span>map</span>(\|</span>name</span>: String, </span>tera</span>: Tera\| { </span> </span>let </span>mut</span> context </span>= </span>Context::new(); </span> context.</span>insert</span>(</span>"name"</span>, </span>&</span>name); </span> </span> </span>let</span> body </span>=</span> tera.</span>render</span>(</span>"hello.html"</span>, </span>&</span>context).</span>unwrap</span>(); </span> warp::reply::html(body) </span> }); </span> </span> warp::serve(hello).</span>run</span>(([</span>127</span>, </span>0</span>, </span>0</span>, </span>1</span>], </span>3000</span>)).await; </span>} </span></code></pre> Now if we test our hello endpoint again, we should see it returning html instead.</p> curl -i http://localhost:3000/hello/jon </span>HTTP/1.1 200 OK </span>content-type: text/html; charset=utf-8 </span>content-length: 126 </span>date: Sat, 04 Jan 2020 19:59:03 GMT </span> </span> </span><!DOCTYPE html> </span><head> </span> <meta charset="UTF-8"> </span> <title>Hello</title> </span></head> </span><body> </span><p>Hello, jon!</p> </span></body> </span></html> </span></code></pre> Or you could point your browser to http://localhost:3000/hello/name, to see the html rendered.</p> Note</strong></p> You will notice that I am using .unwrap()</code> is several places where the function is returning a Result</a>. I am doing that so that I don't have do to error handling yet. I may write another post about error handling but for now, if you want more information about error handling in rust, this section of the book</a> should help.</p> Login and consent</h3> We showed the basic building blocks for creating a web service in the previous section. Let's put that to use and build our integration to our OAuth 2.0 server. We will implement the login and consent pages and their corresponding form post endpoints.</p> We will first need to update dependencies to add Serde</a> which is the library in Rust to serialize/deserialize to various data formats.</p> Cargo.toml</div> ...</span> </span>[dependencies] </span>...</span> </span>serde </span>= { </span>version </span>= </span>"1.0"</span>, </span>features </span>= [</span>"derive"</span>] } </span></code></pre> With this let's implement the login page and form handler first.</p> src/main.rs</div> pub </span>mod </span>view; </span></code></pre> Will create a view module where we can put our template rendering functions.</p> src/view.rs</div> use </span>tera::Tera; </span>use </span>warp::{self, Filter}; </span> </span>pub </span>fn </span>with_tera</span>() -> warp::filters::BoxedFilter<(Tera,)> { </span> warp::any().</span>map</span>(</span>move \|\| </span>tera_templates</span>()).</span>boxed</span>() </span>} </span> </span>pub </span>fn </span>tera_templates</span>() -> Tera { </span> </span>let</span> login_tpl </span>= </span>r</span>#" </span><!DOCTYPE html> </span><head> </span> <meta charset="UTF-8"> </span> <title>Login</title> </span></head> </span><body> </span><h1>Login page</h1> </span><form action="/login" method="post"> </span>{% if login_challenge %} </span> <input type="hidden" name="login_challenge" value="{{login_challenge}}"/> </span>{% else %} </span>{% endif %} </span> </span> <label for="username">Username</label>: </span> <input type="text" id="username" name="username" autofocus="autofocus"/> <br/> </span> </span> <label for="password">Password</label>: </span> <input type="password" id="password" name="password"/> <br/> </span> </span> <input type="submit" value="Log in"/> </span></form> </span></body> </span></html> </span>"#</span>; </span> </span> </span>let </span>mut</span> tera </span>= </span>Tera::default(); </span> tera.</span>add_raw_templates</span>(vec![(</span>"login.html"</span>, login_tpl)]) </span> .</span>unwrap</span>(); </span> tera </span>} </span></code></pre> And now for the actual login page using the rendering template we just created in the view</code> module.</p> src/main.rs</div> use </span>serde::{Deserialize, Serialize}; </span>use </span>tera::{Context, Tera}; </span>use </span>warp::{self, Filter}; </span> </span>pub </span>mod </span>view; </span> </span>#[tokio::main] </span>async </span>fn </span>main</span>() { </span> </span>let</span> routes </span>= </span>auth_routes</span>(); </span> warp::serve(routes).</span>run</span>(([</span>127</span>, </span>0</span>, </span>0</span>, </span>1</span>], </span>3000</span>)).await; </span>} </span>pub </span>fn </span>auth_routes</span>() -> warp::filters::BoxedFilter<(impl warp::reply::Reply,)> { </span> warp::path(</span>"login"</span>) </span> .</span>and</span>(</span>login_page</span>()) </span> .</span>boxed</span>() </span>} </span> </span> </span>pub </span>fn </span>login_page</span>() -> warp::filters::BoxedFilter<(impl warp::Reply,)> { </span> #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] </span> </span>struct </span>LoginQueryParams { </span> login_challenge: </span>Option</span><</span>String</span>>, </span> } </span> </span> warp::get() </span> .</span>and</span>(warp::query::query()) </span> .</span>and</span>(view::with_tera()) </span> .</span>map</span>(</span>move \|</span>query_params: LoginQueryParams, tera: Tera</span>\| </span>{ </span> </span>let </span>mut</span> context </span>= </span>Context::new(); </span> context.</span>insert</span>(</span>"login_challenge"</span>, </span>&</span>query_params.login_challenge); </span> </span> </span>let</span> body </span>=</span> tera.</span>render</span>(</span>"login.html"</span>, </span>&</span>context).</span>unwrap</span>(); </span> warp::reply::html(body) </span> }) </span> .</span>boxed</span>() </span>} </span></code></pre> If we submit the login form from the login page, we should receive a 404. To fix that we need to implement the form post handling endpoint.</p> src/main.rs</div> use </span>serde::{Deserialize, Serialize}; </span>use </span>tera::{Context, Tera}; </span>use </span>warp::{self, Filter}; </span> </span>pub </span>mod </span>view; </span> </span>... </span> </span>pub </span>fn </span>auth_routes</span>() -> warp::filters::BoxedFilter<(impl warp::reply::Reply,)> { </span> warp::path(</span>"login"</span>) </span> .</span>and</span>(</span>login_page</span>().</span>or</span>(</span>accept_login</span>())) </span> .</span>boxed</span>() </span>} </span> </span>... </span> </span>pub </span>fn </span>accept_login</span>() -> warp::filters::BoxedFilter<(impl warp::Reply,)> { </span> #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] </span> </span>struct </span>FormBody { </span> login_challenge: </span>Option</span><</span>String</span>>, </span> username: String, </span> password: String, </span> } </span> </span> warp::post() </span> .</span>and</span>( </span> warp::body::content_length_limit(</span>1024 </span>* </span>32</span>) </span> .</span>and</span>(warp::body::form()) </span> .</span>map</span>(\|</span>form_body</span>: FormBody\| warp::reply::json(</span>&</span>form_body)), </span> ) </span> .</span>boxed</span>() </span>} </span></code></pre> We now have a working login page and login handler which returns the form body back as json. So if you try to login with the username, "username"</code>, and password, "password"</code>, we should receive the following.</p> http://localhost:3000/login</div> { </span> </span>"username"</span>: </span>"username"</span>, </span> </span>"password"</span>: </span>"password"</span>, </span> </span>"login_challenge"</span>: </span>null </span>} </span></code></pre> Following the same pattern, you can add the consent page and post endpoint.</p> You can see what you should end up with here</a></p> Setting up the Hydra services</h3> To be able to test our integration to hydra we will need to run it locally and to do that, the easiest way I found is to run hydra with their docker-compose</a> file. We will then make a few modifications to it. We could run hydra with an in-memory database, but with a local postgres database we will be able to see how the api calls we will be doing to the hydra instance affect its data store.</p> docker-compose.yml</div> version</span>: </span>"3.3" </span>services</span>: </span> </span>postgres</span>: </span> </span>image</span>: </span>postgres:latest </span> </span>ports</span>: </span> - </span>"5432:5432" </span> </span>environment</span>: </span> </span>POSTGRES_PASSWORD</span>: </span>password </span> </span>volumes</span>: </span> - </span>${PWD}/initdb.sh:/docker-entrypoint-initdb.d/init-user-db.sh </span> </span> </span>hydra</span>: </span> </span>depends_on</span>: </span> - </span>hydra-migrate </span> </span>image</span>: </span>oryd/hydra:latest </span> </span>ports</span>: </span> - </span>"4444:4444" </span># Public port </span> - </span>"4445:4445" </span># Admin port </span> - </span>"5555:5555" </span># Port for hydra token user </span> </span>command</span>: </span>serve all --dangerous-force-http </span> </span>environment</span>: </span> </span># https://www.ory.sh/docs/hydra/configuration </span> </span># - LOG_LEVEL=debug </span> - </span>URLS_SELF_ISSUER=http://localhost:4444 </span> - </span>URLS_CONSENT=http://localhost:3000/consent </span> - </span>URLS_LOGIN=http://localhost:3000/login </span> - </span>DSN=postgres://hydra:hello@postgres:5432/hydra?sslmode=disable </span> - </span>SECRETS_SYSTEM=youReallyNeedToChangeThis </span> - </span>SECRETS_COOKIE=youReallyNeedToChangeThisToo </span> - </span>OAUTH2_EXPOSE_INTERNAL_ERRORS=1 </span> - </span>OIDC_SUBJECT_TYPES_SUPPORTED=public,pairwise </span> - </span>OIDC_SUBJECT_TYPE_PAIRWISE_SALT=youReallyNeedToChangeThisToo2 </span> </span>restart</span>: </span>unless-stopped </span> </span> </span>hydra-migrate</span>: </span> </span>depends_on</span>: </span> - </span>postgres </span> </span>image</span>: </span>oryd/hydra:latest </span> </span>environment</span>: </span> - </span>DSN=postgres://hydra:hello@postgres:5432/hydra?sslmode=disable&max_conns=20&max_idle_conns=4 </span> </span>command</span>: </span>migrate sql -e --yes </span> </span>restart</span>: </span>on-failure </span></code></pre> initdb.sh</div> #!/bin/bash </span>set </span>-e </span> </span>psql</span> -v</span> ON_ERROR_STOP=1</span> --username </span>"$POSTGRES_USER"</span> --dbname </span>"$POSTGRES_DB" </span><<-EOSQL </span> CREATE USER hydra WITH PASSWORD 'hello'; </span> CREATE USER auth WITH PASSWORD 'authpassword'; </span> CREATE DATABASE hydra; </span> CREATE DATABASE auth; </span> GRANT ALL PRIVILEGES ON DATABASE hydra TO hydra; </span> GRANT ALL PRIVILEGES ON DATABASE auth TO auth; </span>EOSQL </span></code></pre> The main difference between this docker-compose.yml</code> file and the one provided by in the Hydra repository, is the initdb.sh</code> script that creates 2 databases and the corresponding user and assigns that user to it's database only.</p> Generating the Hydra OpenApi Client</h3> The Hydra server provides public and admin REST APIs</a> to interface with it and allows us to implement the integration. They also provide sdks, for Go and Javascript, but since we are using Rust we will need to implement our own. Luckily they also provide OpenAPI specs</a> and the OpenAPI generator implements a Rust client generator. The code quality isn't great but that should be enough to get us started with our integration and we can always improve it later by implementing our own client using the generated models.</p> Here is the command to generate our rust OpenAPI client.</p> openapi-generator-cli generate -i https://raw.githubusercontent.com/ory/hydra/v1.1.1/docs/api.swagger.json --package-name hydra --library reqwest -g rust -o /local/hydra </span></code></pre> Or even better, instead of installing the openapi-generator-cli</code>, we can use the docker image to do the same thing and let's put it inside a bash script to be able to reuse it if necessary and remember next time we need to regenerate.</p> generate-hydra-api.sh</div> #!/bin/bash </span> </span>set </span>-o</span> errexit </span>set </span>-o</span> pipefail </span>set </span>-o</span> nounset </span>set </span>-x </span> </span>docker run</span> --rm -v </span>${PWD}:/local openapitools/openapi-generator-cli:latest generate</span> -i</span> https://raw.githubusercontent.com/ory/hydra/v1.1.1/docs/api.swagger.json</span> --package-name</span> hydra</span> --library</span> reqwest</span> -g</span> rust</span> -o</span> /local/hydra </span></code></pre> Now we have 2 crates of code, one that we wrote and the other that we generated. We will need to make some space where this new crate can live and compile the together. Cargo provides the workspace functionality for handling multiple crates in the same repository. To do that, we will move the code we have written so far into a sub directory. Let's call that the auth</code> directory.</p> mkdir auth </span>mv Cargo.toml auth </span>mv src auth </span></code></pre> Change the package name in the Cargo.toml</code> to auth to match the directory name.</p> auth/Cargo.toml</div> [package] </span>name </span>= </span>"auth" </span># <--- name to change </span>version </span>= </span>"0.1.0" </span>authors </span>= [</span>"Jonathan Fok Kan <jfokkan@gmail.com>"</span>] </span>edition </span>= </span>"2018" </span> </span>[dependencies] </span>...</span> </span></code></pre> Then create a Cargo.toml</code> file with the 2 sub crates we now have.</p> Cargo.toml</div> [workspace] </span> </span>members </span>= [ </span> </span>"auth"</span>, </span> </span>"hydra"</span>, </span>] </span></code></pre> Lets compile everything and see if the new generated crate works.</p> cargo clean </span>cargo build </span></code></pre> Oh Ho, something's not working. We are getting a compile error.</p> error[E0308]: mismatched types </span> --> hydra/src/apis/admin_api.rs:542:16 </span> \| </span>542 \| if let Some(ref s) = client { </span> \| ^^^^^^^^^^^ ------ this match expression has type `reqwest::Client` </span> \| \| </span> \| expected struct `reqwest::Client`, found enum `std::option::Option` </span> \| </span> = note: expected type `reqwest::Client` </span> found type `std::option::Option<_>` </span> </span>error: aborting due to previous error </span> </span>For more information about this error, try `rustc --explain E0308`. </span>error: could not compile `hydra`. </span>warning: build failed, waiting for other jobs to finish... </span>error: build failed </span></code></pre> Let's check out the file and see what's wrong. The rust compiler is helpful enough to give us the exact location of the error. Well the error looks pretty simple, the client</code> created within the function body at let client = &configuration.client;</code> is shadowing the client</code> from the function parameters. The solution is just as simple, let's just rename the function parameter to client_id</code> since that's actually what is being sent here. The changed lines will be as follows.</p> hydra/src/apis/admin_api.rs</div> ... </span> </span>fn </span>revoke_consent_sessions</span>(</span>&</span>self</span>, </span>subject</span>: </span>&</span>str</span>, </span>client_id</span>: </span>Option</span><</span>&</span>str</span>>) -> </span>Result</span><(), Error> { </span> </span>let</span> configuration: </span>&</span>configuration::Configuration </span>= </span>self.configuration.</span>borrow</span>(); </span> </span>let</span> client </span>= &</span>configuration.client; </span> </span> </span>... </span> </span> req_builder </span>=</span> req_builder.</span>query</span>(</span>&</span>[(</span>"subject"</span>, </span>&</span>subject.</span>to_string</span>())]); </span> </span>if </span>let Some</span>(</span>ref</span> s) </span>=</span> client_id { </span> req_builder </span>=</span> req_builder.</span>query</span>(</span>&</span>[(</span>"client"</span>, </span>&</span>s.</span>to_string</span>())]); </span> } </span> </span> </span>... </span></code></pre> Updating our login and consent implementation</h3> Now that we have our generated hydra client library, we want to integrate this into our auth</code> project. Let's take a look at the code. For reading and getting a quick overview of the code. My preferred way of doing this is by reading the documentation. Even when there is no documentation, Rust generates pretty nice documentation from the public types and provides a fairly intuitive way of navigating and searching the interface. To generate the documentation, Cargo provides the doc</code> command.</p> cargo doc --open </span></code></pre> If your browser didn't open with documentation page, you should be able to find it at ./target/doc/hydra/index.html</code> in the root directory of the project.</p> Once the documentation is opened, you should find the hydra crate which will be in the crates list. Inside, you will find 2 modules, the apis and models modules. Inside the apis module, we will find the structs and traits that will help us communicate with hydra. The AdminApiClient</code> will be the one we care about to implement the login and consent features.</p> impl AdminApiClient </span>pub fn new(configuration: Rc<Configuration>) -> AdminApiClient </span></code></pre> We can see that AdminApiClient</code> has a new function to create a new instance of the struct and it takes a Rc<Configuration></code>. Rc<Configuration></code> is a reference counted pointer to a Configuration struct and the Configuration struct has some fields that helps the ApiClient figure out how to make calls to the hydra instance.</p> Similar to the Tera template renderer, we will create a Warp Filter that will contain the AdminApiClient and pass it to the function handling the endpoints. But first we need to add the dependency to the hydra crate to our main auth crate.</p> auth/Cargo.toml</div> [dependencies] </span>...</span> </span>hydra </span>= { </span>path </span>= </span>"../hydra" </span>} </span>...</span> </span></code></pre> auth/src/main.rs</div> ... </span>use </span>hydra::apis::{configuration::Configuration, AdminApiClient}; </span>use </span>std::rc::Rc; </span> </span>... </span> </span>pub </span>fn </span>with_hydra_api</span>() -> warp::filters::BoxedFilter<(AdminApiClient,)> { </span> warp::any() </span> .</span>map</span>(</span>move \|\| </span>{ </span> </span>let </span>mut</span> configuration </span>= </span>Configuration::new(); </span> configuration.base_path </span>= </span>HYDRA_ADMIN_ADDRESS</span>.</span>to_owned</span>(); </span> AdminApiClient::new(Rc::new(configuration)) </span> }) </span> .</span>boxed</span>() </span>} </span> </span></code></pre> But something's not right, we are getting a compilation error, with rust stating that our Rc<Configuration></code> isn't Send</code>.</p> error[E0277]: `std::rc::Rc<hydra::apis::configuration::Configuration>` cannot be sent between threads safely </span> --> auth/src/main.rs:44:10 </span> \| </span>44 \| .boxed() </span> \| ^^^^^ `std::rc::Rc<hydra::apis::configuration::Configuration>` cannot be sent between threads safely </span> \| </span> = help: within `(hydra::apis::admin_api::AdminApiClient,)`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<hydra::apis::configuration::Configuration>` </span> = note: required because it appears within the type `hydra::apis::admin_api::AdminApiClient` </span> = note: required because it appears within the type `(hydra::apis::admin_api::AdminApiClient,)` </span></code></pre> What is Send? Send</a> is a marker trait that states whether something can be sent between threads (i.e. thread-safe). What can we do? Well not much, it seems that Warp requires state passed into it's filters to be thread safe since it aims to concurrently handle requests with multiple threads but the openapi code generator produced code that can't. We are going to have to modify the generated code just enough to make it thread safe. One way to do that is simply to switch the Rc with Arc</a> which is the thread safe version of Rc.</p> These 2 commands ought to do it.</p> find ./hydra -type f -exec sed -i '' 's/std::rc::Rc/std::sync::Arc/g' {} \; </span>find ./hydra -type f -exec sed -i '' 's/Rc/Arc/g' {} \; </span></code></pre> Now instead of AdminApiClient::new(Rc::new(configuration))</code> we can pass it AdminApiClient::new(Arc::new(configuration))</code>.</p> Let's implement the accept login request logic for the hydra integration in the login page.</p> auth/src/main.rs</div> use </span>log::info; </span>use </span>serde::{Deserialize, Serialize}; </span>use </span>tera::{Context, Tera}; </span>use </span>warp::{self, http::Uri, Filter}; </span> </span>use </span>hydra::apis::{configuration::Configuration, AdminApi, AdminApiClient}; </span>use </span>std::{</span>str</span>::FromStr, sync::Arc}; </span>... </span> </span>pub </span>fn </span>login_page</span>() -> warp::filters::BoxedFilter<(impl warp::Reply,)> { </span> #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] </span> </span>struct </span>LoginQueryParams { </span> login_challenge: </span>Option</span><</span>String</span>>, </span> } </span> </span> warp::get() </span> .</span>and</span>(warp::query::query()) </span> .</span>and</span>(view::with_tera()) </span> .</span>and</span>(</span>with_hydra_api</span>()) </span> .</span>map</span>( </span> </span>move \|</span>query_params: LoginQueryParams, tera: Tera, hydra_api: AdminApiClient</span>\| </span>{ </span> </span> </span>// The challenge is used to fetch information about the login request from ORY Hydra. </span> query_params </span> .login_challenge </span> .</span>map</span>(\|</span>login_challenge</span>\| { </span> </span>let</span> login_request </span>= </span> hydra_api.</span>get_login_request</span>(</span>&</span>login_challenge).</span>unwrap</span>(); </span> </span> </span>// If hydra was already able to authenticate the user, skip will be true and we do not need to re-authenticate </span> </span>if</span> login_request.skip.</span>unwrap_or</span>(</span>false</span>) { </span> info!(</span>"Hydra was already able to authenticate the user, skipping login as we do not need to re-authenticate"</span>); </span> info!(</span>"Accepting login request with Hydra"</span>); </span> </span> </span>// You can apply logic here, for example update the number of times the user logged in. </span> </span>// ... </span> </span> </span>// Now it's time to grant the login request. You could also deny the request if something went terribly wrong </span> </span>// (e.g. your arch-enemy logging in...) </span> </span>let</span> completed_request </span>=</span> hydra_api </span> .</span>accept_login_request</span>( </span> </span>&</span>login_challenge, </span> </span>Some</span>(hydra::models::AcceptLoginRequest::new( </span> </span>// All we need to do is to confirm that we indeed want to log in the user. </span> </span>// We are using a hardcoded subject here, the subject should be an immutable id of the user that is loggin in </span> </span>// to let Hydra know which user to associate with this login. </span> </span>"hardcoded_subject"</span>.</span>to_owned</span>(), </span> )), </span> ) </span> .</span>unwrap</span>(); </span> </span> </span>// All we need to do now is to redirect the user back to hydra! </span> </span>Box</span>::new(warp::redirect( </span> Uri::from_str( </span> </span>&</span>completed_request </span> .redirect_to </span> .</span>unwrap_or</span>(</span>"/"</span>.</span>to_owned</span>()), </span> ) </span> .</span>unwrap</span>(), </span> )) </span>as </span>Box</span><dyn warp::Reply> </span> } </span>else </span>{ </span> </span>// If authentication can't be skipped we MUST show the login UI. </span> info!(</span>"Sending user to login"</span>); </span> </span> </span>// The challenge will be a hidden input field </span> </span>let </span>mut</span> context </span>= </span>Context::new(); </span> context.</span>insert</span>(</span>"login_challenge"</span>, </span>&</span>login_challenge); </span> </span> </span>let</span> body </span>=</span> tera.</span>render</span>(</span>"login.html"</span>, </span>&</span>context).</span>unwrap</span>(); </span> </span>Box</span>::new(warp::reply::html(body)) </span>as </span>Box</span><dyn warp::Reply> </span> } </span> }) </span> .</span>unwrap_or_else</span>(\|\| { </span> </span>let</span> body </span>=</span> tera.</span>render</span>(</span>"login.html"</span>, </span>&</span>Context::new()).</span>unwrap</span>(); </span> </span>Box</span>::new(warp::reply::html(body)) </span>as </span>Box</span><dyn warp::Reply> </span> }) </span> }, </span> ) </span> .</span>boxed</span>() </span>} </span></code></pre> There is quite a bit happening here, so let's explain. First, I added info logging in a few places, which is why I added the log::info</code> dependency. We will need to turn on the logs later on, but let's not worry about that for now. The logic goes as follows:</p> We map the login_challenge</code> from the query parameters, if it doesn't exist, we render a login page without login challenge. I did this because I expected the login page to also be usable outside of an oauth2 flow, in which case we could simply return a session cookie once the user is authenticated. We could also have simply returned an error if the login challenge wasn't there.</li> If the login challenge exists, we call hydra's get login request api and with the login request we check if we could skip the login page if the user had successfully authenticated with this browser in the past.</li> If we can skip the user authentication, we go straight into accepting the login request with the hydra api. This is also where we could have added some logic before calling the accept login api to do various checks.</li> Once the login request is accepted, we redirect to the url provided by hydra or to the homepage if it's not provided. In this case since we are expecting to be redirected by hydra, we could also have returned an error page if the redirect url was empty.</li> If we can't skip authentication, we send the user to the login page with the login challenge set.</li> </ol> Something else you may have noticed is that instead of returning a warp::Reply</code> directly, I wrapped it inside of a Box so that our function is returning a trait object. This is because we are returning 2 different types both implementing the Reply trait in the 2 branches of our if else expression. This wouldn't work unless we make them both the same type by turning them into trait objects. To learn more about Trait Objects check out these links</a>.</p> I also imported the std::str::FromStr</code> trait so that it can be used to convert an &str</code> to a Uri</code>.</p> As for handling the login form, we will be following similar logic as above.</p> auth/src/main.rs</div> ... </span> </span>pub </span>fn </span>accept_login</span>() -> warp::filters::BoxedFilter<(impl warp::Reply,)> { </span> #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] </span> </span>struct </span>FormBody { </span> login_challenge: </span>Option</span><</span>String</span>>, </span> username: String, </span> password: String, </span> } </span> </span> warp::post() </span> .</span>and</span>( </span> warp::body::content_length_limit(</span>1024 </span>* </span>32</span>) </span> .</span>and</span>(warp::body::form()) </span> .</span>and</span>(</span>with_hydra_api</span>()) </span> .</span>map</span>(\|</span>form_body</span>: FormBody, </span>hydra_api</span>: AdminApiClient\| { </span> </span>// Add logic here to verify the username and password from the submitted login form </span> </span> </span>// Accepting login request, although you could still deny the login request if something else went wrong </span> form_body </span> .login_challenge </span> .</span>map</span>(\|</span>login_challenge</span>\| { </span> info!(</span>"Accepting login request with Hydra"</span>); </span> </span>let</span> completed_request </span>=</span> hydra_api </span> .</span>accept_login_request</span>( </span> </span>&</span>login_challenge, </span> </span>Some</span>(hydra::models::AcceptLoginRequest::new( </span> </span>// We are using a hardcoded subject here, the subject should be an immutable id of the user that is loggin in </span> </span>// to let Hydra know which user to associate with this login </span> </span>"hardcoded_subject"</span>.</span>to_owned</span>(), </span> )), </span> ) </span> .</span>unwrap</span>(); </span> </span> </span>// Redirecting to hydra </span> warp::redirect( </span> Uri::from_str( </span> </span>&</span>completed_request.redirect_to.</span>unwrap_or</span>(</span>"/"</span>.</span>to_owned</span>()), </span> ) </span> .</span>unwrap</span>(), </span> ) </span> }) </span> .</span>unwrap_or_else</span>(\|\| warp::redirect(Uri::from_str(</span>"/"</span>).</span>unwrap</span>())) </span> }), </span> ) </span> .</span>boxed</span>() </span>} </span></code></pre> And following the same principles as the login logic, we can do the same for the consent.</p> Enabling logging</h4> Before we finish, let's print the logs that we inserted. To do that we will be using the env_logger library. It's a fairly simple library that allows you to control the log level and which crates should have it's logs enabled through an environment variable, by default that env var is RUST_LOG</code>.</p> I will also add the log Filter from warp to give us access logs on our endpoints.</p> auth/src/main.rs</div> #[tokio::main] </span>async </span>fn </span>main</span>() { </span> </span>if </span>::std::env::var_os(</span>"RUST_LOG"</span>).</span>is_none</span>() { </span> ::std::env::set_var(</span>"RUST_LOG"</span>, </span>"warp=info,auth_svc=trace,api_access=trace"</span>); </span> } </span> env_logger::init(); </span> </span> </span>let</span> routes </span>= </span>auth_routes</span>().</span>or</span>(</span>homepage</span>()).</span>with</span>(warp::log(</span>"api_access"</span>)); </span> warp::serve(routes).</span>run</span>(([</span>127</span>, </span>0</span>, </span>0</span>, </span>1</span>], </span>3000</span>)).await; </span>} </span></code></pre> What you should now have.</a></p> Testing our login and consent integration</h3> We now have working login and consent application and a way to deploy hydra locally. How do we know what we wrote is work? Let's try it out manually and then write some tests for it.</p> docker-compose up -d </span>cargo run </span></code></pre> Create a client in Hydra.</p> curl -X POST \ </span> http://localhost:4445/clients \ </span> -H 'Content-Type: application/json' \ </span> -d '{ </span> "client_id": "my-implicit-client", </span> "grant_types": ["implicit"], </span> "response_types": ["token"], </span> "redirect_uris": ["http://localhost:3000/"], </span> "token_endpoint_auth_method": "client_secret_post" </span>}' </span></code></pre> To start the flow, navigate to Authorization Initialization link</a> which is usually generated by an OAuth2 client or library.</p> After going through the login and consent pages, we should end up back on the welcome page with the access token.</p> http://localhost:3000/#access_token=IQc9NKSJyHS9Vy9iS05kyXVoUTkXexCQPxq-6_Ly5C8.m5m7P-RYhty0o47x35D1uF-k_JJdttzjxqqw11Kr22M&expires_in=3600&scope=offline&state=blahblahblah&token_type=bearer </span></code></pre> Since we know that our integration is working, let's write a test that we can run to verify that it keeps working in the future. To do that, we will create a tests</code> directory in our auth</code> project. That's where integration tests usually go, whereas unit tests can be written inline (For more information on tests check out these links</a>). But we might want to refer to some structs from our crate such as the form model structs, currently we can't do that because we wrote all our code in a main.rs</code> file which is used for creating a binary. We would need to move the code that we would like to be able to share as a library into a lib.rs</code> file.</p> We will add a function that returns our routes which can then be used from the main.rs</code> file.</p> auth/src/lib.rs</div> ... </span>pub </span>fn </span>routes</span>() -> warp::filters::BoxedFilter<(impl warp::reply::Reply,)> { </span> </span>auth_routes</span>() </span> .</span>or</span>(</span>homepage</span>()) </span> .</span>with</span>(warp::log(</span>"api_access"</span>)) </span> .</span>boxed</span>() </span>} </span>... </span></code></pre> auth/src/main.rs</div> use</span> auth; </span>use</span> warp; </span> </span>#[tokio::main] </span>async </span>fn </span>main</span>() { </span> </span>if </span>::std::env::var_os(</span>"RUST_LOG"</span>).</span>is_none</span>() { </span> ::std::env::set_var(</span>"RUST_LOG"</span>, </span>"warp=info,auth_svc=trace,api_access=trace"</span>); </span> } </span> env_logger::init(); </span> </span> warp::serve(auth::routes()) </span> .</span>run</span>(([</span>127</span>, </span>0</span>, </span>0</span>, </span>1</span>], </span>3000</span>)) </span> .await; </span>} </span></code></pre> Then we will have to expose the structs we created to model the form bodies of our login and consent apis.</p> auth/src/lib.rs</div> ... </span>#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] </span>pub </span>struct </span>LoginFormBody { </span> </span>pub </span>login_challenge: </span>Option</span><</span>String</span>>, </span> </span>pub </span>username: String, </span> </span>pub </span>password: String, </span>} </span> </span>... </span> </span>#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] </span>pub </span>enum </span>ConsentStatus { </span> Authorize, </span> Deny, </span>} </span> </span>#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] </span>pub </span>struct </span>ConsentFormBody { </span> </span>pub </span>consent_challenge: String, </span> </span>pub </span>submit: ConsentStatus, </span>} </span>... </span></code></pre> Now that we have a library crate we can get to the actual work of writing the test.</p> auth/Cargo.toml</div> ...</span> </span>[dev-dependencies] </span>uuid </span>= { </span>version </span>= </span>"0.8"</span>, </span>features </span>= [</span>"v4"</span>] } </span>oauth2 </span>= </span>" 3.0.0-alpha.7" </span>reqwest </span>= { </span>version </span>= </span>"0.10.0"</span>, </span>features </span>= [</span>"json"</span>, </span>"blocking"</span>, </span>"cookies"</span>] } </span></code></pre> The body of a basic test is as follows.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> #[test] </span>fn </span>basic_test</span>() { </span> assert!(</span>true</span>); </span>} </span></code></pre> Since this is an integration test we may want to check that our external dependencies are up.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> use </span>reqwest::{self, Url}; </span> </span>#[test] </span>fn </span>login_and_consent_flow</span>() { </span> </span>check_auth_svc</span>(</span>AUTHN_BASE_URL</span>); </span> </span>check_hydra</span>(</span>HYDRA_ADMIN_BASE_URL</span>); </span>} </span> </span>fn </span>check_auth_svc</span>(</span>authn_url</span>: </span>&</span>str</span>) { </span> </span>let</span> res </span>= </span>reqwest::blocking::get(authn_url).</span>unwrap</span>(); </span> assert_eq!(res.</span>status</span>(), reqwest::StatusCode::</span>OK</span>); </span>} </span> </span>fn </span>check_hydra</span>(</span>hydra_url</span>: </span>&</span>str</span>) { </span> </span>let</span> hydra_url </span>= </span>Url::parse(hydra_url).</span>unwrap</span>(); </span> </span>let</span> res </span>= </span>reqwest::blocking::get(hydra_url.</span>join</span>(</span>"/health/ready"</span>).</span>unwrap</span>()).</span>unwrap</span>(); </span> assert_eq!(res.</span>status</span>(), reqwest::StatusCode::</span>OK</span>); </span>} </span></code></pre> As we saw in our manual test, we will first need to create an oauth2 client to do our test. To know which type of oauth2 client we want to create, we should decide what we will be testing. So let's say we will be testing the authorization code grant type.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> use </span>hydra::apis::{configuration::Configuration, AdminApi, AdminApiClient}; </span>use </span>reqwest::{self, Url}; </span>use </span>std::sync::Arc; </span>use </span>uuid::Uuid; </span> </span>#[test] </span>fn </span>login_and_consent_flow</span>() { </span> </span>... </span> </span> </span>let </span>mut</span> config </span>= </span>Configuration::new(); </span> config.base_path </span>= </span>HYDRA_ADMIN_BASE_URL</span>.</span>to_owned</span>(); </span> </span>let</span> hydra_admin_client </span>= </span>AdminApiClient::new(Arc::new(config)); </span> </span> </span>let</span> oauth2_client </span>= </span>create_oauth2_client</span>(</span>&</span>hydra_admin_client); </span>} </span> </span>fn </span>create_oauth2_client</span>(</span>hydra_admin_client</span>: </span>&</span>AdminApiClient) -> hydra::models::OAuth2Client { </span> </span>let </span>mut</span> new_oauth2_client </span>= </span>hydra::models::OAuth2Client::new(); </span> new_oauth2_client.client_id </span>= </span>Some</span>(format!( </span> </span>"</span>{}</span>-</span>{}</span>"</span>, </span> Uuid::new_v4().</span>to_string</span>(), </span> </span>"my-test-client" </span> )); </span> new_oauth2_client.client_name </span>= </span>Some</span>(</span>"login-flow-test-client"</span>.</span>to_owned</span>()); </span> new_oauth2_client.client_secret </span>= </span>Some</span>(</span>"client-secret"</span>.</span>to_owned</span>()); </span> new_oauth2_client.grant_types </span>= </span>Some</span>(vec![</span>"authorization_code"</span>.</span>to_owned</span>()]); </span> new_oauth2_client.redirect_uris </span>= </span>Some</span>(vec![</span>AUTHN_BASE_URL</span>.</span>to_owned</span>()]); </span> new_oauth2_client.token_endpoint_auth_method </span>= </span>Some</span>(</span>"client_secret_basic"</span>.</span>to_owned</span>()); </span> new_oauth2_client.scope </span>= </span>Some</span>(</span>"openid"</span>.</span>to_owned</span>()); </span> </span> hydra_admin_client </span> .</span>create_o_auth2_client</span>(new_oauth2_client) </span> .</span>unwrap</span>() </span>} </span></code></pre> With the created OAuth 2 client, we can initiate an oauth2 flow. For that, we will use an OAuth2 library</a>.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> </span>#[test] </span>fn </span>login_and_consent_flow</span>() { </span> </span>... </span> </span> </span>let</span> access_token </span>= </span>initiate_oauth2_code_flow</span>( </span> </span>&</span>oauth2_client.client_id.</span>unwrap</span>(), </span> </span>&</span>oauth2_client.client_secret.</span>unwrap</span>(), </span> ); </span>} </span> </span>... </span> </span>fn </span>initiate_oauth2_code_flow</span>(</span>client_id</span>: </span>&</span>str</span>, </span>client_secret</span>: </span>&</span>str</span>) -> String { </span> </span>// Create an HTTP client to act as the browser </span> </span>let</span> reqwest_client </span>= </span>reqwest::blocking::Client::builder() </span> </span>// Enable the cookie store for hydra csrf cookies </span> .</span>cookie_store</span>(</span>true</span>) </span> .</span>redirect</span>(reqwest::redirect::Policy::none()) </span> .</span>build</span>() </span> .</span>unwrap</span>(); </span> </span> </span>// Create an OAuth2 client by specifying the client ID, client secret, authorization URL and </span> </span>// token URL. </span> </span>let</span> client </span>= </span>OAuth2BasicClient::new( </span> ClientId::new(client_id.</span>to_owned</span>()), </span> </span>Some</span>(ClientSecret::new(client_secret.</span>to_owned</span>())), </span> AuthUrl::new(</span>"http://localhost:4444/oauth2/auth"</span>.</span>to_owned</span>()).</span>unwrap</span>(), </span> </span>Some</span>(TokenUrl::new(</span>"http://localhost:4444/oauth2/token"</span>.</span>to_owned</span>()).</span>unwrap</span>()), </span> ) </span> </span>// Set the URL the user will be redirected to after the authorization process. </span> .</span>set_redirect_url</span>(RedirectUrl::new(</span>"http://localhost:3000/"</span>.</span>to_string</span>()).</span>unwrap</span>()); </span> </span> </span>// Generate a PKCE challenge. </span> </span>let </span>(pkce_challenge, pkce_verifier) </span>= </span>PkceCodeChallenge::new_random_sha256(); </span> </span> </span>// Generate the full authorization URL. </span> </span>let </span>(auth_url, csrf_token) </span>=</span> client </span> .</span>authorize_url</span>(CsrfToken::new_random) </span> </span>// Set the desired scopes. </span> .</span>add_scope</span>(Scope::new(</span>"openid"</span>.</span>to_string</span>())) </span> </span>// Set the PKCE code challenge. </span> .</span>set_pkce_challenge</span>(pkce_challenge) </span> .</span>url</span>(); </span> </span> </span>// This is the URL you should usually redirect the user to, </span> </span>// in order to trigger the authorization process. </span> println!(</span>"URL to trigger authorization process: </span>\n{}</span>"</span>, auth_url); </span>} </span></code></pre> Usually the authorize url would then be used to redirect the user (resource owner) to initiate the authorization process. But since we are writing an automated test, we will be emulating the browser with an http client and call our login and consent apis directly.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> </span>fn </span>initiate_oauth2_code_flow</span>(</span>client_id</span>: </span>&</span>str</span>, </span>client_secret</span>: </span>&</span>str</span>) -> String { </span> </span>... </span> </span> </span>let</span> hydra_login_completed_url </span>= </span>perform_login_flow</span>(</span>&</span>reqwest_client, </span>&</span>auth_url); </span>} </span> </span>fn </span>perform_login_flow</span>(</span>reqwest_client</span>: </span>&</span>reqwest::blocking::Client, </span>auth_url</span>: </span>&</span>Url) -> String { </span> </span>let</span> hydra_authz_res </span>=</span> reqwest_client.</span>get</span>(auth_url.</span>as_ref</span>()).</span>send</span>().</span>unwrap</span>(); </span> println!(</span>"</span>\n</span>This is the response from Hydra when visiting the url to trigger the authorization process"</span>); </span> dbg!(</span>&</span>hydra_authz_res); </span> </span> assert!(hydra_authz_res.</span>status</span>().</span>is_redirection</span>()); </span> </span> </span>let</span> login_url </span>=</span> hydra_authz_res </span> .</span>headers</span>() </span> .</span>get</span>(reqwest::header::</span>LOCATION</span>) </span> .</span>unwrap</span>() </span> .</span>to_str</span>() </span> .</span>unwrap</span>(); </span> </span>let</span> login_challenge </span>= </span>Url::parse(login_url) </span> .</span>unwrap</span>() </span> .</span>query_pairs</span>() </span> .</span>find</span>(\|</span>c</span>\| c.</span>0.</span>as_ref</span>().</span>eq_ignore_ascii_case</span>(</span>"login_challenge"</span>)) </span> .</span>unwrap</span>() </span> .</span>1 </span> .</span>to_owned</span>() </span> .</span>to_string</span>(); </span> </span> </span>let</span> login_res </span>=</span> reqwest_client </span> .</span>post</span>(</span>"http://localhost:3000/login"</span>) </span> .</span>form</span>(</span>&</span>LoginFormBody { </span> login_challenge: </span>Some</span>(login_challenge), </span> username: </span>"username"</span>.</span>to_owned</span>(), </span> password: </span>"pass"</span>.</span>to_owned</span>(), </span> }) </span> .</span>send</span>() </span> .</span>unwrap</span>(); </span> </span> println!(</span>"</span>\n</span>This is the response from our auth service when visiting the login page"</span>); </span> dbg!(</span>&</span>login_res); </span> </span> assert!(login_res.</span>status</span>().</span>is_redirection</span>()); </span> </span> </span>let</span> hydra_redirected_url </span>=</span> login_res </span> .</span>headers</span>() </span> .</span>get</span>(reqwest::header::</span>LOCATION</span>) </span> .</span>unwrap</span>() </span> .</span>to_str</span>() </span> .</span>unwrap</span>(); </span> </span> hydra_redirected_url.</span>to_owned</span>() </span>} </span></code></pre> You can note that I disabled redirection in our http client so that we can see all the redirects that are occurring. That is not strictly necessary and we could simply have followed the redirects which would also have made our code much shorter.</p> Same thing for the consent.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> </span>fn </span>initiate_oauth2_code_flow</span>(</span>client_id</span>: </span>&</span>str</span>, </span>client_secret</span>: </span>&</span>str</span>) -> String { </span> </span>... </span> </span> </span>let</span> redirected_url </span>= </span>perform_consent_flow</span>(</span>&</span>reqwest_client, </span>&</span>hydra_login_completed_url); </span>} </span> </span>... </span> </span>fn </span>perform_consent_flow</span>( </span> </span>reqwest_client</span>: </span>&</span>reqwest::blocking::Client, </span> </span>hydra_login_completed_url</span>: </span>&</span>str</span>, </span>) -> String { </span> </span>let</span> hydra_consent_res </span>=</span> reqwest_client </span> .</span>get</span>(hydra_login_completed_url) </span> .</span>send</span>() </span> .</span>unwrap</span>(); </span> </span> println!(</span>"</span>\n</span>This is the response from hydra when visiting the url received at the end of the login process"</span>); </span> dbg!(</span>&</span>hydra_consent_res); </span> println!(</span>"</span>\n</span>As we can see, hydra will now redirect us to our consent page since the login request was accepted"</span>); </span> </span> assert!(hydra_consent_res.</span>status</span>().</span>is_redirection</span>()); </span> </span> </span>let</span> consent_url </span>=</span> hydra_consent_res </span> .</span>headers</span>() </span> .</span>get</span>(reqwest::header::</span>LOCATION</span>) </span> .</span>unwrap</span>() </span> .</span>to_str</span>() </span> .</span>unwrap</span>(); </span> </span>let</span> consent_challenge </span>= </span>Url::parse(consent_url) </span> .</span>unwrap</span>() </span> .</span>query_pairs</span>() </span> .</span>find</span>(\|</span>c</span>\| c.</span>0.</span>as_ref</span>().</span>eq_ignore_ascii_case</span>(</span>"consent_challenge"</span>)) </span> .</span>unwrap</span>() </span> .</span>1 </span> .</span>to_owned</span>() </span> .</span>to_string</span>(); </span> </span> </span>let</span> consent_res </span>=</span> reqwest_client </span> .</span>post</span>(</span>"http://localhost:3000/consent"</span>) </span> .</span>form</span>(</span>&</span>ConsentFormBody { </span> consent_challenge: consent_challenge, </span> submit: ConsentStatus::Authorize, </span> }) </span> .</span>send</span>() </span> .</span>unwrap</span>(); </span> </span> println!(</span>"</span>\n</span>This is the response from our auth service when visiting the consent page"</span>); </span> dbg!(</span>&</span>consent_res); </span> </span> assert!(consent_res.</span>status</span>().</span>is_redirection</span>()); </span> </span> </span>let</span> hydra_consent_approved_url </span>=</span> consent_res </span> .</span>headers</span>() </span> .</span>get</span>(reqwest::header::</span>LOCATION</span>) </span> .</span>unwrap</span>() </span> .</span>to_str</span>() </span> .</span>unwrap</span>(); </span> </span> </span>let</span> hydra_consent_approved_res </span>=</span> reqwest_client </span> .</span>get</span>(hydra_consent_approved_url) </span> .</span>send</span>() </span> .</span>unwrap</span>(); </span> println!(</span>"</span>\n</span>This is the response from hydra once the consent request was accepted. You can note that it contains the authorization code appended to the redirect url configured at the start of the OAuth2 process"</span>); </span> dbg!(</span>&</span>hydra_consent_approved_res); </span> </span> </span>let</span> redirected_url </span>=</span> hydra_consent_approved_res </span> .</span>headers</span>() </span> .</span>get</span>(reqwest::header::</span>LOCATION</span>) </span> .</span>unwrap</span>() </span> .</span>to_str</span>() </span> .</span>unwrap</span>(); </span> </span> redirected_url.</span>to_owned</span>() </span>} </span></code></pre> With the final redirected url, we will receive an auth code that is used in the authorization code grant type to get our access token.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> </span>fn </span>initiate_oauth2_code_flow</span>(</span>client_id</span>: </span>&</span>str</span>, </span>client_secret</span>: </span>&</span>str</span>) -> String { </span> </span>let</span> auth_code </span>= </span>Url::parse(</span>&</span>redirected_url) </span> .</span>unwrap</span>() </span> .</span>query_pairs</span>() </span> .</span>find</span>(\|</span>c</span>\| c.</span>0.</span>as_ref</span>().</span>eq_ignore_ascii_case</span>(</span>"code"</span>)) </span> .</span>unwrap</span>() </span> .</span>1 </span> .</span>to_owned</span>() </span> .</span>to_string</span>(); </span> </span> </span>// Once the user has been redirected to the redirect URL, you'll have access to the </span> </span>// authorization code. For security reasons, your code should verify that the `state` </span> </span>// parameter returned by the server matches `csrf_state`. </span> </span>// Here we are skipping this check for the purposes of brevity of this test. </span> </span> </span>// Now you can trade it for an access token. </span> </span>let</span> token_result </span>=</span> client </span> .</span>exchange_code</span>(AuthorizationCode::new(auth_code.</span>to_string</span>())) </span> </span>// Set the PKCE code verifier. </span> .</span>set_pkce_verifier</span>(pkce_verifier) </span> .</span>request</span>(http_client) </span> .</span>unwrap</span>(); </span> println!(</span>"</span>\n</span>Once the authorization code is exchanged, we finally received an access token"</span>); </span> dbg!(</span>&</span>token_result); </span> println!(</span>"</span>{:?}</span>"</span>, token_result.</span>access_token</span>().</span>secret</span>()); </span> token_result.</span>access_token</span>().</span>secret</span>().</span>to_owned</span>() </span>} </span></code></pre> Now that we have an access token, we should probably see if we can verify it with the introspection endpoint</a>.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> #[test] </span>fn </span>login_and_consent_flow</span>() { </span> </span>... </span> </span> </span>let</span> access_token </span>= </span>initiate_oauth2_code_flow</span>( </span> </span>&</span>oauth2_client.client_id.</span>unwrap</span>(), </span> </span>&</span>oauth2_client.client_secret.</span>unwrap</span>(), </span> ); </span> </span> </span>introspect_access_token</span>(</span>&</span>hydra_admin_client, </span>&</span>access_token); </span>} </span> </span>... </span> </span>fn </span>introspect_access_token</span>(</span>hydra_admin_client</span>: </span>&</span>AdminApiClient, </span>access_token</span>: </span>&</span>str</span>) { </span> </span>let</span> introspection_res </span>=</span> hydra_admin_client </span> .</span>introspect_o_auth2_token</span>(access_token, </span>None</span>) </span> .</span>unwrap</span>(); </span> dbg!(</span>&</span>introspection_res); </span> assert!(introspection_res.active); </span>} </span></code></pre> We can now run our test to see everything running. Since I added some print statements, to be able to see those we will also need to run the test command with the --nocapture</code> flag.</p> cargo test -- --nocapture </span></code></pre> Code with the test</a></p> Async Rust</h3> If you have kept up to date with Async Rust, you may have noticed that the hydra generated client uses the blocking version of the Reqwest library to make it's api calls. This means that by making these API calls we are blocking the tokio executor. What this means is that in cases of load on our auth server, the server may have all it's executor threads blocked on the blocking call we are currently making and prevent the server from serving as many requests as it could.</p> There are 2 ways to fix, the easiest would be to move the blocking call into a thread where blocking isn't an issue. To do that we could spawn the blocking call on a seperate thread where we wouldn't block the tokio executor. See these links for more info</a>.</p> The second would be to rewrite the blocking api client with a non-blocking http client. The latest version of reqwest is such a client.</p> Where to go from here?</h2> We now have a working OAuth 2.0 authorization server integrated with a bare bones auth server. Having a working OAuth server is only the start of implementing a good authorization system for a web service. Integrating it with your services, making access control decisions, how to pass authorization information to your resource servers are only some of the things that still need to be done to get a working system. We also took several shortcuts in this demo to keep the code short and focus on the parts we care about more but there are also several things that need to be done to make our service production ready. Here are several things I can see need improvement in no particular order:</p> Error handling, currently our service will panic at anything deviating from the happy path</li> Add CSRF protection to our login and consent pages and endpoints. This is necessary to prevent a class of attacks called Login CSRF</a></li> Add User management features to our auth service so that it can implement authentication logic</li> Add actual authentication logic to the login endpoints</li> Deploy our service to kubernetes. Hydra already provides Helm Charts for that</a></li> Making sure to follow Hydra's production guide before deploying our services. Link</a></li> Put our services behind an API Gateway or reverse proxy so that all our services are available at one address</li> Add https certificates to our services, this may be easier with a reverse proxy/API Gateway. This is necessary because OAuth 2 simply doesn't have any protection without it</li> Protect a web service's resources(a resource server) with our authorization server</li> Integrate with Ory's other projects such as Keto</a> to provide Access Control rules processing</li> </ul> References</h2> </div> RFC 6749: The OAuth 2.0 Authorization Framework</a></span></li> List of IETF OAuth 2.0 RFCs</a></span></li> Links to various documents about OAuth 2.0</a> from oauth.net</span></li> The latest security best practices for OAuth 2.0</a></span></li> OAuth 2.0 Simplified</a> by Aaron Parecki</span></li> OAuth 2.0 OpenID Connect</span> OpenID Connect Explained</a></li> An OpenID Connect Primer from Okta Identity, Claims, & Tokens: Part 1</a></li> OIDC In Action: Part 2</a></li> What's in a Token?: Part 3</a></li> </ol> </li> </ol> </li> A previous Hydra integration I wrote with Spring and Java Git Repo</a></span></li> Information about when you may want to use different types of access control (Access Tokens vs session cookies)</a></span></li> Ory Hydra REST API docs</span> Ory Hydra REST API docs</a></li> OryOS.14 Versioned Github link</a></li> </ol> </li> OpenAPI</span> OpenAPI Generator</a></li> Github Link</a></li> </ol> </li> Rust Send and Sync Traits</span> Send std docs</a></li> Send and Sync from the Rustonomicon</a></li> </ol> </li> Rust Trait Objects</span> Generic Data Types</a></li> Trait Objects</a></li> </ol> </li> Rust Testing</span> Writing Automated Tests from the Rust Book</a></li> Testing from Rust By Example</a></li> </ol> </li> OAuth 2.0 Token Introspection Extension</span> OAuth2 Token Introspection Endpoint from oauth.com</a></li> RFC 7662: OAuth 2.0 Token Introspection</a></li> Ory Hydra Introspection docs</a></li> </ol> </li> Async Rust: Tokio Blocking</span> tokio::task</a></li> tokio::task::block_in_place</a></li> tokio::task::spawn_blocking</a></li> </ol> </li> </ol> Go and JSON, various useful libraries and utilities Unknown — Fri, 12 Aug 2016 00:00:00 +0000 JSON is probably one of the most popular data interchange format on the web. Go with its extensive standard library has good support for JSON. If your requirements for JSON serialization and deserialization are not specific, I highly recommend to simply start by using "encoding/json"</code> from the standard library and reading how it works (blog</a>, docs</a>, examples</a>). Here I will give a few libraries for JSON in Go and explain what differentiates them from "encoding/json" and each other.</p> First, why would we want to use anything other than "encoding/json"</code>? Well to provide flexibility and simplicity in its usage, "encoding/json"</code> makes use of reflection on the structs that defines the structure of the JSON you are expecting to serialize or deserialize and because of that it incurs a heavy performance penalty.</p> What if we didn't have to reflect on a struct and simply insert data to a map[string]interface{}</code> object? Well "encoding/json"</code> allows you to do that but the resulting data structure ends up frustrating to work with, with casting all over the place and is hard to navigate fluently.</p> Structured Data</h2> ffjson</a></li> easyjson</a></li> </ul> To avoid reflection, these libraries provide a code generator to be run over structs defining the structure of the JSON and fields in the same format as "encoding/json"</code>. The generated code is now the parsing code for the JSON structure.</p> Both ffjson</code> and easyjson</code> provide a drop in replacement for "encoding/json"</code> which means that after generating code for parsing, you should be able to use the same api calls as before but with ffjson</code> or easyjson</code>.</p> The advantage of this is that you now have specific parsing code for the JSON structure you are expecting. This also gives a nice speed up in the order of about 2 to 3 times faster</strong> than encoding/json</code>.</p> The disadvantage is that you will need to run the code generator every time you change the struct defining the JSON structure. There is a simple fix for this by using the go generate</code> command and //go:generate ffjson $GOFILE</code> at the top of any files with the structs.</p> easyjson</code> works very similarly to ffjson</code> and claims to be more performant and behave better when run concurrently.</p> Deserializing Unstructured Data</h2> jsonparser</a></li> gjson</a></li> </ul> jsonparser</code> and gjson</code> like ffjson</code> and easyjson</code> do not rely on reflection but what differentiates them from the structured JSON libraries is that instead of creating structs that defines the structure of the JSON to be parsed these libraries read the value at a specific path you want.</p> By only reading values at paths specified, you only need to parse the part of the data you are interested in. No need to parse the whole json eagerly. Users of lazy functional languages will recognize here the value of lazy reading. This allows these libraries to read even faster than ffjson</code> and easyjson</code>.</p> By not having to deserialize to a struct, these libraries can read directly from the byte buffer which allows values to be read much faster than other deserialization libraries.</p> Another advantage over "encoding/json"</code>, is that the api is much more ergonomic than unmarshalling the json into a map[string]interface{}</code> since no casting is necessary and walking to the specific field you want to read is much easier. For example, here is how to read values using "encoding/json"</code> with map[string]interface{}</code> vs gjson</code>.</p> // "encoding/json" </span>package </span>main </span> </span>import </span>( </span> </span>"encoding/json" </span>) </span> </span>const </span>data </span>= </span>`{"name":{"first":"Janet","last":"Prichard"},"age":47}` </span> </span>func </span>main</span>() { </span> </span>var </span>values </span>map</span>[</span>string</span>]</span>interface</span>{} </span> json.Unmarshal([]</span>byte</span>(data), </span>&</span>values) </span> value </span>:= </span>values[</span>"name"</span>].(</span>map</span>[</span>string</span>]</span>interface</span>{})[</span>"last"</span>].(</span>string</span>) </span> printf(value) </span>} </span> </span>// gjson </span>package </span>main </span> </span>import </span>"github.com/tidwall/gjson" </span> </span>const </span>json </span>= </span>`{"name":{"first":"Janet","last":"Prichard"},"age":47}` </span> </span>func </span>main</span>() { </span> value </span>:= </span>gjson.Get(json, </span>"name.last"</span>) </span> </span>println</span>(value.String()) </span>} </span></code></pre> The disadvantage of these libraries is that they only provide deserialization and not serialization. For fast serialization, you can rely on the structured data serializers.</p> Utilities and other helpful libraries</h2> Convert JSON to a Go Struct</h3> Instead of doing this job by hand, here are some utilities that can help you:</p> Online: https://mholt.github.io/json-to-go/</li> Locally: https://github.com/ChimeraCoder/gojson</li> </ul> Summary</h2> Useful JSON libraries: For generating encoders and decoders:</p> ffjson</a></li> easyjson</a></li> </ul> For reading unstructured json at a specific path:</p> jsonparser</a></li> gjson</a></li> </ul> Spice up your LaTeX with Font Awesome Unknown — Tue, 02 Feb 2016 00:00:00 +0000 LaTeX allows you to create amazing documents which are beautifully typeset. It gives you access to a wide breath of tools and graphics to improve your documents. One of these are the many already available symbols in LaTeX. Here is a comprehensive list of available symbols: Link</a> PDF</a>. You can even find symbols by drawing them at this service</a>. These include mathematical symbols, phonetic symbols, currency symbols or symbols for various languages.</p> I found out recently that you can include Font Awesome</a> icons as symbols in your latex documents using this CTAN package</a>. This allows you to use some modern icons such as the Github or LinkedIn logos to spice up your documents such as a resume if you write it in LaTeX. If you have a full TeX live distribution such as MacTeX, it should be already included. I was able to use it by adding \usepackage{fontawesome}</code> to my document and using the lualatex</code> command to compile my document. I wasn't able figure out why xelatex</code> failed to compile my document. If you do let me know.</p> The final result:</p> </p> You can checkout the source here: Link</a></p> A simple protocol for a TCP chat server and client Unknown — Sun, 21 Jun 2015 00:00:00 +0000 Introduction</h2> I will be talking about some motivating factors for why you would want to use a lower level networking protocol such as TCP and how you could implement your own application protocol over it.</p> Some prerequisites</h3> a general understanding of networking</li> a Go distribution</li> </ul> Motivation</h2> Dealing with an HTTP API such as a RESTful web API is often the easier way to interact with your server. Applications that only need to consume and send commands to the server asynchronously can fit this use case. But some applications such as a chat client or a game server need a 2-way method of communication between the client and server. This is accomplished by using a lower level protocol such as TCP or UDP.</p> We can think of TCP and UDP as a pipe between 2 endpoints and at each end of the connection we have a stream of bytes that can be read. I will note here that TCP is in this analogy a reliable pipe whereas we can consider UDP as a faster but unreliable pipe between the endpoints. Further detail about the differences of TCP and UDP are beyond the scope of this post, but I encourage you to do some research to learn more since networking has some very leaky abstractions nad understanding the layer underneath is always useful in debugging issues such as outages, latency and any properties you would like to have.</p> How can we use tcp</h2> Now that we know what we can use to have a continuous stream of data to communicate between 2 endpoints, how can we actually used that stream of bytes? We first need a way to interpret that stream of bytes so that we can read the messages passed on there.</p> There are several ways to do it:</p> We could use a set number of bytes</li> We could use sentinel values to mark the beginning and/or end of a message</li> Something else(let's come back to this)</li> </ul> The pros and Cons</h3> Pros of using a set number of bytes:</p> Size of buffer to allocate for the message does not change and is known in advance</li> Data in the message can be arbitrary and does not need to be escaped</li> </ul> Cons of using a set number of bytes:</p> Can be wasteful. What if we need to send 8 bytes but we are using 1024 byte buffers</li> Hard to extend. Once we declare the protocol we cannot change the number of bytes in messages</li> </ul> Pros of using sentinel values</p> Easy to implement. Read until we see the sentinel value.</li> Flexible in how much data we can send.</li> </ul> Cons of using sentinel values</p> We cannot use the sentinel values in our data. We need to carefully escape the data to be sent.</li> </ul> Is there a better option?</h2> Glad you asked.</p> We could define a protocol that specifies how much data to be sent by using the set number of bytes scheme. This essentially ends up being a header. We could define a protocol that sets the number of bytes for the header which always needs to be sent followed by the message body.</p> A simple example of using this scheme would be to have a 4 byte message for a 32bit int which would describe the length of the message followed by the message. This will allow us to allocate a buffer of that size and read until we receive all the bytes promised in the header.</p> This negates the wastefulness of the fixed number of bytes scheme, but it still makes it hard to extend this protocol without breaking backwards compatibility. For example, in our simple example protocol, we can send a maximum of 4GB in our messages. (4 bytes = 48 = 32 bits; 2^32-1 = 4294967295 bytes = 4.29497 GB)</p> Lets use this to implement a simple chat server and client</h2> Using this scheme we can wrap the net.Conn</code> Read</code> method in a helper function as follows:</p> func </span>ReadMsg</span>(</span>conn </span>net.</span>Conn</span>) (</span>string</span>, </span>error</span>) { </span> </span>// Make a buffer to hold length of data </span> lenBuf </span>:= </span>make</span>([]</span>byte</span>, </span>4</span>) </span> _, err </span>:= </span>conn.Read(lenBuf) </span> </span>// Receiving EOF means that the connection has been closed </span> </span>if </span>err </span>== </span>io.EOF { </span> </span>// Close conn and exit </span> conn.Close() </span> fmt.Println(</span>"Connection Closed. Bye bye."</span>) </span> os.Exit(</span>0</span>) </span> } </span> </span>if </span>err </span>!= </span>nil </span>{ </span> </span>return </span>""</span>, err </span> } </span> lenData </span>:= </span>FromBytes(lenBuf) </span> </span> </span>// Make a buffer to hold incoming data. </span> buf </span>:= </span>make</span>([]</span>byte</span>, lenData) </span> reqLen </span>:= </span>0 </span> </span>// Keep reading data from the incoming connection into the buffer </span> </span>// until all the data promised is received </span> </span>for </span>reqLen </span>< </span>int</span>(lenData) { </span> tempreqLen, err </span>:= </span>conn.Read(buf[reqLen:]) </span> reqLen </span>+= </span>tempreqLen </span> </span>if </span>err </span>== </span>io.EOF { </span> </span>return </span>""</span>, fmt.Errorf(</span>"Received EOF before receiving all promised data."</span>) </span> } </span> </span>if </span>err </span>!= </span>nil </span>{ </span> </span>return </span>""</span>, fmt.Errorf(</span>"Error reading: </span>%s</span>"</span>, err.Error()) </span> } </span> } </span> </span>return </span>string</span>(buf), </span>nil </span>} </span></code></pre> What this does is read 4bytes from the specified connection and converts those bytes to an int. It then reads from the connection until it has seen as many bytes from the connection as was promised in the header. The FromBytes</code> function takes care of converting the bytes to the int. In our protocol, the int are encoded in binary in Big Endian as is common in most networking protocol which is why big endian is also known as The Network Byte Order</em>. Here is the helper functions to convert ints to and from bytes:</p> // To convert Big Endian binary format of a 4 byte integer to int32 </span>func </span>FromBytes</span>(</span>b </span>[]</span>byte</span>) </span>int32 </span>{ </span> buf </span>:= </span>bytes.NewReader(b) </span> </span>var </span>result </span>int32 </span> err </span>:= </span>binary.Read(buf, binary.BigEndian, </span>&</span>result) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> </span>return </span>result </span>} </span> </span>// To convert an int32 to a 4 byte Big Endian binary format </span>func </span>ToBytes</span>(</span>i </span>int32</span>) []</span>byte </span>{ </span> buf </span>:= </span>new</span>(bytes.</span>Buffer</span>) </span> err </span>:= </span>binary.Write(buf, binary.BigEndian, i) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> </span>return </span>buf.Bytes() </span>} </span></code></pre> Similarly for the write:</p> func </span>WriteMsg</span>(</span>conn </span>net.</span>Conn</span>, </span>msg </span>string</span>) { </span> </span>// Send the size of the message to be sent </span> conn.Write([]</span>byte</span>(ToBytes(</span>int32</span>(</span>len</span>([]</span>byte</span>(msg)))))) </span> </span>// Send the message </span> conn.Write([]</span>byte</span>(msg)) </span>} </span></code></pre> We write the size of the message and then the message.</p> Now lets use these in a server:</p> func </span>main</span>() { </span> </span>// Listen for incoming connections. </span> l, err </span>:= </span>net.Listen(CONN_TYPE, CONN_HOST</span>+</span>":"</span>+</span>CONN_PORT) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> fmt.Println(</span>"Error listening:"</span>, err.Error()) </span> os.Exit(</span>1</span>) </span> } </span> </span>// Close the listener when the application closes. </span> </span>defer </span>l.Close() </span> fmt.Println(</span>"Listening on " </span>+ </span>CONN_HOST </span>+ </span>":" </span>+ </span>CONN_PORT) </span> </span>for </span>{ </span> </span>// Listen for an incoming connection. </span> conn, err </span>:= </span>l.Accept() </span> </span>if </span>err </span>!= </span>nil </span>{ </span> fmt.Println(</span>"Error accepting: "</span>, err.Error()) </span> os.Exit(</span>1</span>) </span> } </span> </span>// Handle connections in a new goroutine. </span> </span>go </span>handleRequest(conn) </span> } </span>} </span> </span>// Handles incoming requests. </span>func </span>handleRequest</span>(</span>conn </span>net.</span>Conn</span>) { </span> </span>// Close the connection when you're done with it. </span> </span>defer </span>conn.Close() </span> </span> msg, err </span>:= </span>common.ReadMsg(conn) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> fmt.Printf(</span>"Message Received: </span>%s\n</span>"</span>, msg) </span> </span>// Send a response back to person contacting us. </span> common.WriteMsg(conn, </span>"Message Received."</span>) </span>} </span></code></pre> Here we setup the server to listen for connections at a given port and handle the connection when we receive one. You will notice we are closing the connection as soon as we send a reply, but this is not necessary and we could listen for several messages depending on your application.</p> On the client side:</p> func </span>main</span>() { </span> tcpAddr, err </span>:= </span>net.ResolveTCPAddr(</span>"tcp"</span>, CONN_HOST</span>+</span>":"</span>+</span>CONN_PORT) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> </span> </span>// Connect to server through tcp. </span> conn, err </span>:= </span>net.DialTCP(</span>"tcp"</span>, </span>nil</span>, tcpAddr) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> </span>defer </span>conn.Close() </span> </span>go </span>printOutput(conn) </span> writeInput(conn) </span>} </span> </span>func </span>writeInput</span>(</span>conn </span></span>net.</span>TCPConn</span>) { </span> fmt.Println(</span>"Enter text:"</span>) </span> </span>for </span>{ </span> </span>// Read from stdin. </span> reader </span>:= </span>bufio.NewReader(os.Stdin) </span> fmt.Print(</span>"> "</span>) </span> text, _ </span>:= </span>reader.ReadString(</span>'\n'</span>) </span> common.WriteMsg(conn, text) </span> } </span>} </span> </span>func </span>printOutput</span>(</span>conn </span></span>net.</span>TCPConn</span>) { </span> </span>for </span>{ </span> </span> msg, err </span>:= </span>common.ReadMsg(conn) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> fmt.Println(msg) </span> } </span>} </span></code></pre> We first connect to the server and the same port our server is listening to and set a goroutine to print all messages we receive from the connection, if the connection is closed we exit. On another thread of execution we listen for input on stdin and write messages to the connection.</p> And Voila. We have a client chatting with our server.</p> Concluding remarks</h2> You will probably notice we are not handling the errors and simply exit on failure on any errors in this example. Don't do that in your code. I just wanted to keep my example simple here. In an actual program, we would return err in the function that has one and the top-level would handle them.</p> You can now extend this example to send more meaningful things that just strings. Interesting things such as XML or JSON or Protobuf.</p> The full example is available HERE</a></p> Experiments with Processing Unknown — Fri, 08 May 2015 00:00:00 +0000 Processing is a programming language targeted at electronic arts and visual design communities. It can serve as the foundation for electronic sketchbooks and aims to teach it's target communities programming in a visual context. At its core processing is a simple programming language with a very lightweight C-style syntax, similar to Java in which it is implemented.</p> There are now several implementations of Processing such as the one used on this page using Processing.js</a>. The original one is still the easiest one to use in my opinion but I haven't given a try to the others very thoroughly.</p> The context</h2> I am moving to a new apartment soon and in the process of buying new furniture I have to map out the dimensions of my apartment. I could do a simple paper sketch or simply draw the map in a paint program but my inner geek compels me to try out something new. I have tried several declarative visual programming languages but I haven't found any that I was comfortable with. It may simply be because I am just not made for visual design but I wanted to give this another chance.</p> I tried D3.js for data visualizations and a few graphing libraries in javascript but after trying Processing again, I can state that Processing seems to have the simplest semantics as well as being the most flexible in what it can do. It even has 3D visualizations!</p> Without much more ado here is the map of my apartment.</p> The Map:</h2> </iframe> PFont f; </span> </span>void setup() { </span> size(1000, 800); </span> f = createFont("Arial",16,true); </span>} </span> </span>void draw() { </span> textFont(f,12); </span> // start with the door </span> line(10, 300, 400, 300); </span> line(10, 300, 10, 310); </span> rect(10, 310, 3, 30); </span> line(10, 340, 10, 350); </span> text("Front Door",11,325); </span> line(10,350,300,350); </span> text("258x115",125,320); </span> </span> // kitchen </span> line(260,300, 260,80); </span> line(260,80, 400,80); </span> line(400,80, 400,300); </span> rect(315, 300, 30,3); </span> text("Kitchen", 300, 200); </span> text("170", 410,190); </span> </span> // kitchen column </span> line(400,275, 380,275); </span> line(380,300, 380,275); </span> </span> // bathroom </span> line(220,350, 220,500); </span> line(220,500, 300, 500); </span> line(300,500, 300,350); </span> rect(245,350, 30,3); </span> </span> // bedroom </span> line(300,350, 300,640); </span> line(300,640, 500,640); </span> line(500,640, 500,350); </span> text("Bedroom", 370, 470); </span> text("208",380,625); </span> text("287", 475,500); </span> </span> // living room </span> line(400,80, 835,80); </span> line(835,80, 835,350); </span> line(835,350, 500,350); </span> text("Living room", 550, 200); </span> text("435", 570, 95); </span> text("305", 805, 170); </span> text("142", 800, 345); </span> text("235",600,345); </span> </span> // tv stand </span> rect(720,350, 60,6); </span> text("TV Stand", 750, 370); </span> </span> // balcony door </span> rect(835,100, 3,30); </span> </span> // window </span> rect(835,180, 4,150); </span>} </span></code></pre> Moving from markdown to asciidoc Unknown — Sat, 11 Apr 2015 00:00:00 +0000 Markdown is probably the most popular markup language in the tech community. Often used in comment sections, blogs (such as jekyll), readmes on github or even documentation. Its popularity can easily be explained by its simplicity and extensibility, but this simplicity is both it's advantage and it's biggest weak point. Markdown is so simple that most of its users often extend it into a dialect and that's why there are so many versions of markdown (github markdown, stackoverflow markdown, etc...). If you are interested in markdown, it is worthwhile to check out CommonMark</a> which is a positive step in trying to standardize markdown and specify many of its extensions.</p> I have been using markdown for a few years and unless I am writing a long form document with rich formatting, I am often writing in markdown. It is very readable, easy and quick to write, and the fact that github renders markdown are the main reasons for why I use it, but I have been noticing some problems when it comes to writing slightly longer documents with structure such as notes. Breaking out LaTeX would not be very productive, so I found a solution: asciidoc</a>.</p> Asciidoc is also a readable document format which has equivalents for any of markdowns constructs and more. Things such as table of contents, continuous blocks can be grouped in elements, etc. It feels just as simple as markdown with much more power for richer formatting if needed. I have already converted some of my documents over to asciidoc and the process has gone swimmingly well.</p> Here is a simple readme in asciidoc:</p> = Readme </span>Author Name <name@email.com> </span> </span>== Heading 1 </span> </span>[source,haskell] </span>file.hs </span>---- </span>collatz 1 = [1] </span>collatz n </span> \| even n = n:chain (n `div` 2) </span> \| odd n = n:chain (3n + 1) </span>---- </span> </span>=== list of stuff: </span>* Lorem ipsum dolor sit amet, consectetur </span>* Donec lorem purus, vulputate et elementum tempus </span>* Ut blandit sagittis arcu, sit amet </span></code></pre> Some quick references while writing your asciidoc documents:</p> Asciidoc quick reference</a></li> [http://asciidoctor.org/docs/user-manual](Asciidoctor User Manual)</li> </ul> What I learned from Functional Programming Principles Unknown — Mon, 23 Feb 2015 00:00:00 +0000 Functional Programming Principles is a course taught at Ecole polytechnique fédérale de lausanne and on coursera by Prof. Martin Odersky.</p> I originally took the class to get familiar with scala before tackling the more interesting class by Martin 'Principles of Reactive Programming' which is about asynchronous computation and distributed computation. I had already had an intro to functional programming through SML in the COMP302 class taught at McGill by the brilliant Prof. Brigitte Pientka.</p> The class starts with a model of evaluation for functional programming that is a useful mental model of how to think about functional programs. It is interesting to note that SICP does a similar introduction to programming which I think is better than the usual hello world program many intro cs classes start with.</p> The rest of the class is about functional programming in scala and the various data structures available in the extensive scala standard library, such as lists, sets, maps and streams. The type hierarchy of the collections library is as follows:</p> +--Iterable </span> +--Seq </span> +--IndexedSeq </span> +--Vector </span> +--(Array) Imported from java does not subclass IndexSeq but </span> +--(String) Use wrappers to behave like IndexedSeq </span> +--LinearSeq </span> +--List </span> +--Set </span> +--Map </span></code></pre> Martin also when through some inductive proofs on properties of lists and trees. He also showed how for-expressions can be mapped to map and flatMap and can be more readable in many situations. The course ended with how to operate on lazy collections such as streams.</p> I liked the course overall as it was a good refresher on functional programming as well and a way to gain familiarity to Scala which is probably a factor for why Scala is gaining popularity in some circles. Scala is an interesting language which mixes object oriented programming with functional programming in a way that feels very natural.</p> You can see my notes on the course at : Link</a></p> Some notes on bash scripting Unknown — Fri, 20 Feb 2015 00:00:00 +0000 In your day to day tasks you often encounter tasks that can be easily automated. Instead of wasting time doing menial or mechanical tasks, you could be applying your brain power to automating them. If you are a developer, this is doubly relevant to you to manage your processes, build, cleaning your project directory, etc.</p> Tasks such as massaging data from one format to another or quickly searching and replacing words in certain files, can easily be dealt with a scripting language. One scripting language that is widely available on most modern Linux and Unix systems is bash. Many people will argue to write in sh which is described in the POSIX standard, but we won't be going into this debate here. Instead I can state that most of these tips will be relevant for both.</p> Why not Python, Perl or Ruby?</h2> Although I believe these more fully featured language are appropriate for many tasks even beyond simple scripts, I think it is still a useful skill to learn to script with your shell.</p> I have been doing some devops works recently and had to use a lot of shell scripting to glue together the continuous integration pipeline. I don't claim to be a bash expert but here are some tips I picked up.</p> Don't hardcode the path to bash in your shebang</h2> Instead use #!/usr/bin/env bash</code>. This allows your script to be more portable since bash is not always located under /bin/bash in all linux distros.</p> Always use set -e</h2> When running most scripts that are not simple on off jobs, you may want to be notified of an error occurring and stop the execution of the script. By using "set -e" at the top of your script, it will prevent the execution of the rest of your job if something fails. This allows you to be explicit when you can allow some command to fail which is rarer than when you expect all the commands to run.</p> set </span>-e </span></code></pre> Use set -x for better debugging and logs</h2> When running a script whose output will be captured such as in a ci pipeline, it is useful to know what command was running last when something occurred, such as when an exception occurred. Pretty useful for logs for this same reason.</p> set </span>-x </span></code></pre> Use getopts for easy command line options</h2> Instead of parsing the command line parameters manually you can simply use the getopts library to do the job. It supports only short command line options to make things simpler.</p> #!/usr/bin/env bash </span> </span>while </span>getopts </span>"h:f"</span> OPTION</span>; do </span> </span>case </span>$OPTION </span>in </span> h</span>) </span> </span>echo </span>"usage" </span> ;; </span> f</span>) </span> FILE</span>=</span>$OPTARG </span> </span>echo </span>$FILE </span> ;; </span> </span>esac </span>done </span></code></pre> A column ':' before a character tells getopts which character needs a value. A column ':' at the beginning of the optstring (e.g. ":h:s") sets OPTION var with “?” and the $OPTARG with the wrong character and no output to stderr. Otherwise it would print an illegal argument message to stderr, if an invalid flag is used.</p> Print a multiline string</h2> Instead of printing with several echos for each line use here documents.</p> cat </span><< EOF </span>usage: dotfiles [-h][-u][-p][-d] </span> </span>syncs dotfiles between ~/dotfiles and ~/ </span> </span>OPTIONS: </span> -h print usage </span> -u updates dotfiles in ~/ </span> -p pushes changes to dotfiles in ~/ to ~/dotfiles </span> -d view the difference between dotfiles in ~/ and ~/dotfiles </span>EOF </span></code></pre> Lexing and parsing in Go (Part 1) Unknown — Wed, 04 Feb 2015 00:00:00 +0000 Note: all the code can be seen at github.com/jonfk/calc</a></em></p> About a month ago, I watched a video of Rob Pike</a> giving a talk about lexing for the text/template/parse package. In it he described the simplicity and elegance of making a traditionally not concurrent problem into a concurrent solution. This then motivated me to try out making a lexer for a yet undefined language of my own design. But first as I did in my implementation, let start with the lexer.</p> Lexical Analysis</h2> My quest started simply with a lexer. The idea behind the lexer is actually quite simple. The lexer has a goroutine that runs the lexer and another that collects the tokens sent by the lexer. This setup allows the lexer to be run in a very simple manner. Instead of having mutually recursive functions, we have a for loop that runs functions set by the lexer. The for loop is as follows:</p> // stateFn represents the state of the scanner as a function </span>// that returns the next state. </span>type </span>stateFn </span>func</span>(</span></span>Lexer</span>) </span>stateFn </span> </span>// run runs the state machine for the lexer. </span>func </span>(</span>l </span></span>Lexer</span>) </span>run</span>() { </span> </span>for </span>l.state </span>= </span>lexStart; l.state </span>!= </span>nil</span>; { </span> l.state </span>= </span>l.state(l) </span> } </span>} </span></code></pre> As you will notice, l is a pointer to the lexer struct and lexStart is the first function passed to the loop. The state field of the lexer holds a state function which is a function with the above type signature. The state function returns another state function which will take care of the next state and so on.The for loop then runs the current state until a state function returns nil. The run() function is ran in a goroutine and communicates with the main goroutine through a channel</p> // lex creates a new scanner for the input string. </span>func </span>Lex</span>(</span>name</span>, </span>input </span>string</span>) </span></span>Lexer </span>{ </span> l </span>:= &</span>Lexer{ </span> name: name, </span> input: input, </span> items: </span>make</span>(</span>chan Token</span>), </span> } </span> </span>go </span>l.run() </span> </span>return </span>l </span>} </span></code></pre> In the other goroutine, users of the lexer can make calls to NextItem() which returns the next Token in the channel.</p> Tokens</h2> At this point, you would be right to wonder what exactly are we lexing, since we cannot lex tokens we don't know. I started with the code from the text/template/parse package</a> and modified it to suit my needs. Tokens are defined as a struct with fields for the type, position and value of the token.</p> // token represents a token or text string returned from the scanner. </span>type </span>Token </span>struct </span>{ </span> Typ </span>TokenType </span>// The type of this token. </span> Pos </span>Pos </span>// The starting position, in bytes, of this item in the input string. </span> Val </span>string </span>// The value of this item. </span>} </span></code></pre> The TokenType is then an int defined with the const iota specs. e.g.</p> const </span>( </span> ERROR </span>TokenType </span>= </span>iota </span>// error occurred; value is text of error </span> BOOL </span>// boolean constant </span> EOF </span> NEWLINE </span>// '\n' </span> </span> </span>... </span> </span> </span>// Keywords appear after all the rest. </span> KEYWORD </span>// used only to delimit the keywords </span> ELSE </span>// else keyword </span> END </span>// end keyword </span> IF </span>// if keyword </span> THEN </span>// then keyword </span> LET </span>// let keyword </span> VAR </span>// var keyword </span> VAL </span>// val keyword </span> </span> OPERATOR </span> </span>// Operators and delimiters </span> ADD </span>// + </span> SUB </span>// - </span> MUL </span>// </span> QUO </span>// / </span> REM </span>// % </span> </span> </span>... </span>) </span></code></pre> The TokenType is mapped to their string representation using a map.</p> var </span>key </span>= </span>map</span>[</span>string</span>]</span>TokenType</span>{ </span> </span>"else"</span>: ELSE, </span> </span>"end"</span>: END, </span> </span>"if"</span>: IF, </span> </span>"then"</span>: THEN, </span> </span>"let"</span>: LET, </span> </span>"val"</span>: VAL, </span> </span>"var"</span>: VAR, </span> </span>"+"</span>: ADD, </span> </span>"-"</span>: SUB, </span> </span>"*"</span>: MUL, </span> </span>"/"</span>: QUO, </span> </span>"%"</span>: REM, </span> </span>"&&"</span>: LAND, </span> </span>"\|\|"</span>: LOR, </span> </span>"=="</span>: EQL, </span> </span>"<"</span>: LSS, </span> </span>">"</span>: GTR, </span> </span>"="</span>: ASSIGN, </span> </span>"!"</span>: NOT, </span> </span>"!="</span>: NEQ, </span> </span>"<="</span>: LEQ, </span> </span>">="</span>: GEQ, </span>} </span></code></pre> A few other helper functions and methods are defined such as precedence(), IsLiteral(), Equals(), etc.</p> Next we will talk about the ast.</p> How I setup my emacs config Unknown — Sun, 21 Dec 2014 00:00:00 +0000 Last Edited 04 Feb 2015</em></p> First some background. I used vim for about 2 years. It was my first real editor back when I was a first year CompSci student. It was a hard road in the beginning, learning modal editing, the quirks of vim's configuration and various useful plugins using Tim Pope's pathogen</a>.</p> After a few years of vim usage, you could wonder why I switched, especially since I go out of my way to set up emacs to replicate how I was using vim. Extensibility is the main reason. In emacs, the ease of extending the capabilities of your editor to suit your needs makes my switch worthwhile. I haven't given a real try to elisp yet (that's on my todo list) but binding a function to a keymap is much easier to do and extending the functionality of your editor through it's API is a breeze.</p> One of the well known sayings about emacs is : "a great operating system, lacking only a decent editor". So although the default editor that comes with emacs is not great, you can make it your</i> best editor. The following packages are what makes emacs a better editor for me.</p> EVIL</h2> The most important piece of my emacs config is evil-mode. Evil brings modal editing to emacs and replicates many of vim's features. All the keybindings I was used to in vim are there. I don't claim to be the most advanced vim user out there. There are many who use vim's advanced features in such a way that would prevent them from switching, but if your main reason to stay with vim is modal editing, I urge you to give evil a try.</p> package.el</h2> Another reason why I prefer emacs is that since emacs24, package.el comes on the default installation. This means that I can bootstrap my emacs from it's config file and have my customized environment in a matter of seconds.</p> Evil Leader</h2> Although not as essential as it's namesake mode, evil-leader is useful to prevent rsi and map many of emacs obscure keybindings to a simple 2 keypress. For example I map my leader to the ,</code> key and kill-buffer</code> is mapped to k, so to kill the current buffer I do ,-k</code></p> My config is linked here: .emacs</a></p>

Tool</th>	Strength</th>	Pain Point</th></tr></thead>
Telescope / Snacks pickers</strong></td>	Fast fuzzy search</td>	Still bounce between multiple pickers for keymaps, commands, help, and settings. And you kind of need to know what you are looking for.</td></tr>
`:map</code>, :imap</code>, :verbose map {lhs}</code></strong></td>`	Raw lists of mappings</td>	Zero context—hard to see why</em> and where</em> a mapping was set</td></tr>
which-key.nvim</strong></td>	Gorgeous pop-ups</td>	Can’t enumerate insert-mode</strong> maps or show a fully-flattened view</td></tr>
Plugin docs</strong></td>	Detailed reference</td>	Scattered across dozens of Markdown files; you must already know which plugin to look at. A lot to read through when you are looking for something in particular.</td></tr> </tbody></table> Put together, that’s a lot of jumping around and context switching.</p> My Configs So Far</h2> I have been using kickstart.nvim</strong> for a while now and this rabbit hole began when I decided it was time to upgrade my own config</strong> and figured I’d see what other distros were doing right. I tried out LazyVim, LunarVim, NvChad, and friends.</p> Kickstart</h3> You read the entire config and own it. I still had blind spots (the lspconfig</code> / Mason interactions), but I still felt like I knew my config and where to change something if I needed to. Because the expected use is to pull the config and start modifying it for your own use, I found upgrading your current kickstart config can get difficult. You have to handle merge conflicts, so of which can include major refactorings and outright changes to the plugins used by the config. This means that an upgrade is more than simply resolving merge conflicts to options, you might need to be ready to understand the config anew.</p> `This is why I found it easier to simply restart the config whenever I am upgrading kickstart. It's actually a pretty nice process since on every upgrade you understand neovim much more and can make deeper and more specific changes.</p>` Full-featured distros</h3> A bunch of Lua wires everything up, but these distros don't usually expect the average user to read through them. You’re meant</em> to override their configs to customize them, yet finding the right lever is a scavenger hunt. (I will admit that only LazyVim got a real trial. I only played around a bit with LunarVim)</p> Bottom line</strong>: distro exploration feels like spelunking without a headlamp.</p> Exploration Fatigue</h2> It might very well be a skill issue on my part but I find that exploring a config to find out keymaps, commands and other configured settings quite scattered and not straightforward at all.</p> Quick demo: find where</em> a mapping is defined</strong></p> I use this when I've spotted a mapping but don’t know which plugin or file created it.</p> " Normal-mode maps that start with <leader>f </span>:verbose </span>nmap </span><leader></span>f </span>" Output ends with something like: </span>" Last set from /path/to/plugin/file.lua line 42 </span></code></pre> When I am exploring things generally, this is what I do.</p> " List all normal-mode keymaps via Telescope </span>:Telescope keymaps mode=n </span> </span>" Get every mapping (across all modes) and dump to a scratch buffer </span>:</span>lua</span> vim</span>.</span>print</span>(vim</span>.</span>tbl_map</span>(</span>function</span>(m) </span>return</span> m</span>.</span>lhs </span>end</span>, vim</span>.</span>api</span>.</span>nvim_get_keymap</span>(</span>''</span>))) </span> </span>" Fuzzy-find every help tag—great for plugin docs </span>:Telescope help_tags </span> </span>" Jump straight to the help page for a specific plugin (e.g. Telescope) </span>:help telescope</span>.</span>nvim </span>" (Most plugins expose a tag that matches their repo name.) </span></code></pre> Each of these helps, but none gives me the single dashboard I’m after.</p> A Better Approach?: A Queryable Neovim</h2> Imagine chatting with your editor:</p> “What keymaps are active right now?” “Who defines gx, and where?” “List all Telescope customisations.”</p> Simply dumping config files into an LLM hits token limits on big distros. A more scalable path is to expose runtime APIs an LLM can call:</p> list installed plugins</li> fetch help for any command/topic</li> enumerate active keymaps with their sources</li> </ul> With that, an MCP-style RPC server could feed structured answers to the model. There’s already a promising skeleton: bigcodegen/mcp-neovim-server</a> aimed at editing assistance; its architecture looks adaptable for this my use-case. I might be able to modify it to do what I want.</p> </i> What’s MCP? Model-Control-Protocol: a lightweight JSON-RPC layer that lets an LLM plugin talk to Neovim over stdio or TCP, request data, and send actions back.</p> </div> </div> Have you hacked on something similar, or know plugins I’ve missed? Drop me a note. I’d love to swap ideas and maybe prototype this together.</p> Bye tmux, hello neovim. How I am trying neovim as my terminal multiplexer Unknown — Sun, 27 Apr 2025 00:00:00 +0000 I've been using Tmux as my terminal multiplexer for a little bit and well I have been pretty satisfied with the setup that I have. I have a set of plugins that allow me to do some pretty things I've been wanting to try a Tmux free workflow to that end I have tweaked my NeoVim config to allow Neovim to become the main central point of my development workflow. Now that means that instead of my terminal being the central point that hosts all of the other programs and processes to do various things in my workflow, I'm trying to replace that with Neovim. Let's see how this goes.</p> But before explaining how I'm going to do this with Neovim, let's take a short detour how I'm currently using Tmux as my terminal multiplexer.</p> My Tmux config and workflow</h1> You can find my full unedited tmux config at this link: 70e3965</a>.</p> Tmux Sessions</h2> My main workflow revolves around creating sessions for each project I am working on. I can then switch between sessions my this little fzf command. I find the experience much better than the default choose tree. You can fuzzy find the session you are looking for in a few keys real quick.</p> bind s display-popup -E "tmux list-sessions \| sed -E 's/:.$//' \| grep -v \"^$(tmux display-message -p '#S')\$\" \| fzf --reverse --preview 'tmux capture-pane -ep -t {}' --preview-window=right:60% \| xargs tmux switch-client -t" </span></code></pre> Vi Copy Mode</h2> For getting text in and out of a tmux buffer, I use the vi</code> copy-mode. It allows me to use vi keybindings to navigate, yank, paste, etc.</p> # vi bindings for copy mode </span>setw -g mode-keys vi </span>bind -T copy-mode-vi v send -X begin-selection </span>bind -T copy-mode-vi y send-keys -X copy-pipe-and-cancel </span>bind -T copy-mode-vi MouseDragEnd1Pane send-keys -X copy-pipe-and-cancel </span></code></pre> Tmux windows in sessions and workflow</h2> A little quality of life thing I have is to start tmux windows at 1 instead of at 0. That's because it keeps all the window numbers close together even if you don't have a lot of windows. Not a super big thing, but helps a lot I find.</p> # Start windows and panes at 1, not 0 </span>set -g base-index 1 </span>setw -g pane-base-index 1 </span></code></pre> While we are here, let's talk about how I create windows for each session. Like many other vimmers, my first window in a tmux session, is a Neovim window. The next window (2), is always a shell prompt. It is always free for interactive use. This allows me to go back and forth between the shell and nvim.</p> Then the next few other buffers are project dependent, but a few common ones are a file watcher that runs and rebuilds the project as files change, documentation, an ssh session, etc.</p> I am a 1 window, 1 screen kind of person. While I have experimented with splits and multiple screens in the past, I found that there would always be 1 main window at which I would always staring at, and a bunch of other things on the other screens that I would really rarely care about. And if I tried to make more effective use of the other screens, I would be moving my head around way too much and giving me neck strain. Overall, I found my comfort best with 1 screen on which I concentrate on, and being able to switch between the various windows or tabs effectively and quickly. How I do that quickly and effectively, is program and OS dependent. I might expand on that one day. For now you can take a quick look at my hammerspoon config</a> if you want to know how I do that for things outside of tmux and nvim.</p> Tmux plugins</h2> Finally, here's the shortlist of plugins that I use. I try to keep it fairly short since I generally don't like to have to remember too many different things from various plugins. Let's go through them.</p> set -g @plugin 'tmux-plugins/tpm' </span>set -g @plugin 'tmux-plugins/tmux-sensible' </span>set -g @plugin 'catppuccin/tmux#v2.1.1' </span>set -g @plugin 'tmux-plugins/tmux-yank' </span>set -g @plugin 'christoomey/vim-tmux-navigator' </span>set -g @plugin 'tmux-plugins/tmux-resurrect' </span>set -g @plugin 'tmux-plugins/tmux-continuum' </span></code></pre> TPM</strong> is the Tmux Plugin Manager</a>. It's the easiest way to manage plugins. Although it is a bit annoying that I need to manually fetch and initialize it before it can do it's work. Maybe one day it could come default with a tmux install.</li> tmux-sensible</strong> is a small package with a bunch of default settings that are pretty sensible. I used to carry around a bunch of such settings and found that that package was pretty much what I wanted and cleaned up my config.</li> catppuccin</strong> is nice for the eyes.</li> tmux-yank</strong> allows to yank to the system clipboard which makes integrating with other applications much easier.</li> vim-tmux-navigator</strong> gives keybinds for switching between tmux panes and nvim panes easy. I honestly used it very little because I rarely do splits in tmux.</li> tmux-resurrect</strong> and tmux-continuum</strong> for session persistence. Resurrect restarts sessions from the previous instance of tmux and continuum autosaves the sessions so I don't have to think about it.</li> </ol> What am I going to do in Neovim?</h1> So as I mentioned earlier, I basically plan on making NeoVim my terminal multiplexer. So instead of starting a terminal instance and then starting Tmux sessions inside of that terminal instance and in those Tmux sessions start the NeoVim instances and various other shell prompts. I will start a Neovim instance and from there start terminal buffers inside of Neovim.</p> From my light amount of testing of this workflow, I found that NeoVim's terminal works pretty well for me and my zsh config. So I'm pretty happy with zsh in Neovim.</p> As for how I will replicate sessions into Neovim, well Neovim</a> has :mksessions</code> which allows you to create sessions and I'll be using a few plugins to basically do the same thing as what I was doing with tmux-resurrect and tmux-continuum which is to restart sessions whenever the tmux instance was killed or in this case the Neovim instance gets killed and autosave the sessions as I am using them.</p> I am currently trying out the auto-session</strong> plugin. You can see the changes to my config here: commit</a>.</p> For terminal buffers, I can start them with :terminal</code> and I can switch between the buffers as I did before with telescope. I can even rename the terminal buffers with :file new-buffer-name</code> if I ever need longer running instances.</p> One thing that I would like to figure out would be to have tmux like persistence of the shell history when restarting the sessions and the ability to restart the previously running processes in the terminal buffers if the session is killed. But for now these are minor concerns that I will tackle when the time comes.</p> Now, in closing, I would like to mention that what allows me to do this is the fact that I make a fairly lightweight use of Tmux, other than being able to yank things from the buffers, being able to save sessions and restart sessions automatically, and basically a nice little visual flair with catpucchin. And I think those can be fairly easily replicated inside of neovim. This may not be without issues for people who make much heavier use of split spans and various other power features of tmux but since that's not me I'm gonna give it a try.</p> A Git Configuration Overhaul Unknown — Tue, 25 Feb 2025 00:00:00 +0000 I was drinking my morning coffee while browsing the usual tech updates when I stumbled across an article</a> about how core Git developers configure their own git config</code> settings. This immediately caught my attention, since I was curious to learn how the people who actually work on Git configure their own setups. It seemed like valuable insider knowledge.</p> My Spartan Config Before</h2> My Git configuration was pretty minimal before reading this article:</p> [user] </span> </span>name </span>=</span> Jonathan Fok kan </span> </span>email </span>=</span> jonathan</span>@fokkan.</span>ca </span>[core] </span> </span>editor </span>=</span> nvim </span>[alias] </span> </span>review </span>= </span>"!f(){ git fetch $1 pull/$2/head;git checkout FETCH_HEAD;}; f" </span> </span>shove </span>= </span>"!f(){ git push --force-with-lease $1 $2;}; f" </span>[includeIf "gitdir:~/work/"] </span> </span>path </span>= .</span>gitconfig</span>.</span>work </span></code></pre> The only somewhat advanced thing I was using was the includeIf</code> directive, which allows different configurations for personal versus work projects and different SSH keys for each context.</p> What I Learned</h2> The article detailed settings that even core Git developers change from the defaults. It turns out there are numerous quality-of-life improvements that could be made with just a few configuration tweaks.</p> One of the linked articles</a> about git autosquash</code> was particularly intriguing. At a high level, autosquash allows you to create "fixup" commits that automatically get squashed into the right place during an interactive rebase. Instead of manually reordering commits during a rebase, you can mark a commit as a fixup for a previous commit, and Git will handle the placement automatically. This would make one of my common Git workflows much more efficient, though I'll need to experiment with it more before it fully clicks.</p> A video showing how autosquash works</summary> </iframe> </details> From the Hacker News comments, I discovered Delta</a>, a syntax-highlighting pager for Git. I've implemented it, and while I appreciate the syntax highlighting, I find the UI a bit busy. I'm curious to see how it displays three-way diffs with zdiff3</code>, but I might need some time to get used to it compared to the plain diff view I'm accustomed to.</p> I also learned that adding -p</code> to various Git commands like log</code> and add</code> will show the actual changes, which is super useful.</p> My Expanded Config After</h2> After reading the article and comments, I've significantly expanded my Git configuration:</p> [user] </span> </span>name </span>=</span> Jonathan Fok kan </span> </span>email </span>=</span> jonathan</span>@fokkan.</span>ca </span>[core] </span> </span>editor </span>=</span> nvim </span>[alias] </span> </span>review </span>= </span>"!f(){ git fetch $1 pull/$2/head;git checkout FETCH_HEAD;}; f" </span> </span>shove </span>= </span>"!f(){ git push --force-with-lease $1 $2;}; f" </span> </span>out </span>= </span>"log @{u}.." </span>[column] </span> </span>ui </span>=</span> auto </span>[branch] </span> </span>sort </span>= -</span>committerdate </span>[tag] </span> </span>sort </span>=</span> version</span>:</span>refname </span>[init] </span> </span>defaultBranch </span>=</span> main </span>[diff] </span> </span>algorithm </span>=</span> histogram </span> </span>colorMoved </span>=</span> plain </span> </span>mnemonicPrefix </span>= </span>true </span> </span>renames </span>= </span>true </span>[push] </span> </span>default </span>=</span> simple </span> </span>autoSetupRemote </span>= </span>true </span> </span>followTags </span>= </span>true </span>[fetch] </span> </span>prune </span>= </span>true </span> </span>pruneTags </span>= </span>true </span> </span>all </span>= </span>true </span>[help] </span> </span>autocorrect </span>=</span> prompt </span>[commit] </span> </span>verbose </span>= </span>true </span>[rerere] </span> </span>enabled </span>= </span>true </span> </span>autoupdate </span>= </span>true </span>[rebase] </span> </span>autoSquash </span>= </span>true </span> </span>autoStash </span>= </span>true </span> </span>updateRefs </span>= </span>true </span>[pull] </span> </span>rebase </span>= </span>true </span>[core] </span> </span># a matter of taste (uncomment if you dare) </span> </span># Useful for really large repos but adds the overhead of a process per repository </span> </span># fsmonitor = true </span> </span># untrackedCache = true </span>[core] </span> </span>pager </span>=</span> delta </span>[interactive] </span> </span>diffFilter </span>=</span> delta </span>--</span>color</span>-</span>only </span>[delta] </span> </span>navigate </span>= </span>true </span># use n and N to move between diff sections </span> </span>dark </span>= </span>true </span># or light = true, or omit for auto-detection </span>[merge] </span> </span># (just 'diff3' if git version < 2.3) </span> </span>conflictstyle </span>=</span> zdiff3 </span></code></pre> Some highlights of what I've added:</p> Better branch and tag sorting (branch.sort</code>, tag.sort</code>)</li> Improved diff algorithm and visualization (diff.algorithm</code>, colorMoved</code>)</li> Enhanced pushing and fetching (push.autoSetupRemote</code>, fetch.prune</code>)</li> Rebase improvements (rebase.autoSquash</code>, rebase.autoStash</code>)</li> Delta integration for prettier diffs</li> out</code> alias for listing all unpushed commits Read it with the Terminator's voice</li> </ul> </li> </ul> I'm particularly looking forward to seeing how the zdiff3</code> conflict style helps with merge conflicts, as this was highly recommended by the core developers.</p> While I haven't enabled the filesystem monitoring options (fsmonitor</code> and untrackedCache</code>), I might consider them for larger repositories despite the overhead of running additional processes.</p> It's amazing how much I was able to improve my Git experience with just these configuration changes. Sometimes it pays to see how the experts configure their own tools!</p> Here is a link to my dotfiles repo if you are interested. There might have been more updates since. Github</a></p> Sources</h2> How Core Git Developers Configure Git</a></li> HN comments for "How Core Git Developers Configure Git"</a></li> Fixing up Git with Autosquash</a></li> </ul> My keyboard symbol layer Unknown — Mon, 24 Feb 2025 00:00:00 +0000 Background</h1> I am a dvorak keyboard user. I learned to touch type fairly late in life, in university at the age of 18 or so. Before then, I had my own idiosyncratic way of touch typing with 2 or 3 fingers and glancing to the keyboard for certain keys. To improve my typing accuracy and speed while doing my CS degree, I tried to learn it on qwerty but had a hard time because I already had habits that were hard to break. To break those habits and learn from a clean slate, I ended up learning dvorak.</p> Aside: Dvorak was the popular alternative keyboard layout at the time, while there is much more in this space now, I think dvorak is a decent choice since I noticed a lot of support from various software at all levels of the stack from games to OSes. But the level of support might not really be necessary now with custom keyboards.</p> QMK & Co</h1> I tried the Planck not that long ago. The ortholinear layout promised vast improvements for homerowing because all the keys are parallel, it's easier to move around. At least that was the promise. I found it hard to get used to, but still persevered. What stopped me was the wrist pain I started getting. I am not sure if it was the ortholinear or the higher profile, but in the end I had to abandon it.</p> What really got me on it though was the QMK software, which allows you to customize the keyboard at the firmware level. That was pretty incredible because you now don't need to have OS or software support. All the customization you would want can be done on the keyboard itself. There are so many options for customizations, I would recommend to take a look for yourself if that interests you. A few in particular I found useful were the extra layers, home row mods, and tap dancing.</p> Layers</h1> By default, keyboards are actually already layered. If you squint, the shift key is a layer trigger where each key now has another character it can type. So layers are essentially a way to get extra keys under certain circumstances, the simplest one being when a key is pressed and held such as with shift.</p> My customizations</h1> I currently type on a regular apple magic keyboard. I like it a lot mainly because it's so similar to the Macbook Pro keyboard which I am often using. I find the typing experience pretty comfortable.</p> Because it's a regular keyboard, I don't have many options for layer keys. But because of a few of my own idiosyncracies, I don't use the mod keys to the right of the spacebar much. So I am now experimenting with using the key to the right of the spacebar as my symbol layer trigger.</p> The idea of the symbol layer is to map some of the special characters that I find hard to type for some reason. I can't exactly explain why but moving my right pinky and ring finger is hard and I often mistype those symbols. In addition to that, I find that moving my right hand to reach the number row is slower and I often have a hard time settling my fingers back to the homerow.</p> So to help with that I decided to remap the special characters that are on the right side of the keyboard while keeping the characters that I am currently content with. I decided to add a small ux improvement that the symbol trigger also acts as the shift for certain keys so that I can more easily access them. The reason for those is because I often favor the left shift and therefore end up typing those left side symbols with one hand instead of 2 as would be the recommended way.</p> Without much further ado, here are the layouts.</p> </p> Layout on keyboard-layout-editor</a></p> If you prefer plain text</summary> # Standard 60%/65% Apple Keyboard </span> </span>## QWERTY Base layer </span> </span>┌─────┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───────┐ </span>│ ` │ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │ 8 │ 9 │ 0 │ - │ = │ BKSP │ </span>├─────┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─────┤ </span>│ TAB │ Q │ W │ E │ R │ T │ Y │ U │ I │ O │ P │ [ │ ] │ \ │ </span>├───────┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴─────┤ </span>│ CAPS │ A │ S │ D │ F │ G │ H │ J │ K │ L │ ; │ ' │ RETURN │ </span>├────────┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴────────┤ </span>│ SHIFT │ Z │ X │ C │ V │ B │ N │ M │ , │ . │ / │ SHIFT │ </span>├──────┬───┴┬──┴──┬┴───┴───┴───┴───┴───┴──┬┴───┴┬──┴─┬────────┤ </span>│ LWIN │LALT│ LCTL│ SPACE │ RCTL│RALT│ Arrows │ </span>└──────┴────┴─────┴───────────────────────┴─────┴────┴────────┘ </span> </span>## Dvorak Base layer </span> </span>┌─────┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───────┐ </span>│ ` │ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │ 8 │ 9 │ 0 │ [ │ ] │ BKSP │ </span>├─────┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─────┤ </span>│ TAB │ ' │ , │ . │ P │ Y │ F │ G │ C │ R │ L │ / │ = │ \ │ </span>├───────┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴─────┤ </span>│ CAPS │ A │ O │ E │ U │ I │ D │ H │ T │ N │ S │ - │ RETURN │ </span>├────────┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴────────┤ </span>│ SHIFT │ ; │ Q │ J │ K │ X │ B │ M │ W │ V │ Z │ SHIFT │ </span>├──────┬───┴┬──┴──┬┴───┴───┴───┴───┴───┴──┬┴───┴┬──┴─┬────────┤ </span>│ LWIN │LALT│ LCTL│ SPACE │ RCTL│RALT│ Arrows │ </span>└──────┴────┴─────┴───────────────────────┴─────┴────┴────────┘ </span> </span>## Dvorak Symbol layer </span> </span>┌─────┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───────┐ </span>│ │ ! │ @ │ # │ $ │ % │ ^ │ & │ │ ( │ ) │ { │ } │ BKSP │ </span>├─────┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─────┤ </span>│ TAB │ " │ < │ > │ │ │ │ │ │ │ │ / │ = │ \| │ </span>├───────┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴┬──┴─────┤ </span>│ CAPS │ [ │ ] │ ( │ ) │ │ ^ │ & │ * │ / │ = │ │ RETURN │ </span>├────────┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴─┬─┴────────┤ </span>│ SHIFT │ : │ │ { │ } │ │ │ │ │ │ │ SHIFT │ </span>├──────┬───┴┬──┴──┬┴───┴───┴───┴───┴───┴──┬┴───┴┬──┴─┬────────┤ </span>│ LWIN │LALT│ LCTL│ SPACE │ RCTL│RALT│ Arrows │ </span>└──────┴────┴─────┴───────────────────────┴─────┴────┴────────┘ </span></code></pre> </details> Implementation</h1> I am currently only implementing this on xremap for my linux desktop for now. If I find it useful, I will add it to my karabiner config too.</p> One note is that because I am using the dvorak keyboard layout and then doing the remaping for the symbol layer on xremap, the config is a bit weird. If you are interested in something similar, you would have to adapt it to your specific setup.</p> It might actually be simpler if I did the dvorak keyboard remapping directly in xremap. Not sure of what the side effects of that would be, but something to experiment with possibly.</p> xremap config</a></p> NFS vs SMB transfer speed on a Linux server Unknown — Tue, 08 Oct 2024 00:00:00 +0000 I also posted the full data in a git repo at https://github.com/jonfk/nfs-smb-benchmarking-experiment</a>.</p> NFS vs SMB transfer speed on a Linux server</h1> I have been setting up my homelab server, migrating it from baremetal to Proxmox, a virtualization solution. The whole process is probably a blog post for another time. But while doing the migration, I got bitten by the improvement bug, where I started wanting to improve various features I had decided on a good enough solution at the time. There are probably several blog posts worth of things to discuss just on what I did, but let's not get side tracked.</p> One of the things was using SMB vs NFS on Linux. I have a Synology (ahem, actually two) NAS which I use for storing large amounts of data that are not actively used on the system that generates it or the data might need to be shared through the filesystem with multiple devices. Some examples are backups, photos, videos, entertainment media, etc. In this particular case, it was a backup share from my Synology DS218+ to this new VM I was setting up.</p> So in the past, I had tried to use an NFS mount from my Linux server to the NAS but kept hitting issues with permissions and user mappings that I didn't care to resolve at the time. The setup for SMB was simple enough that I decided to just go with it at the time. But the original reason I wanted to try NFS was that it is supposed to be more resource efficient and theoretically faster than SMB. So I finally decided to barrel through all the issues and got it working. Now with 2 working mounting solutions, I wanted to see if I could benchmark the difference.</p> So here is what I found out. But first, how I ran the tests and some info about the servers.</p> Setup</h1> Server</h2> A Synology DS218+ NAS with the following specs:</p> </th> </th></tr></thead> CPU Model</strong></td> Intel Celeron J3355</td></tr> CPU Architecture</strong></td> 64-bit</td></tr> CPU Frequency</strong></td> Dual Core 2.0 burst up to 2.5 GHz</td></tr> Hardware Transcoding Engine</strong></td> H.264 (AVC), H.265 (HEVC), MPEG-2 and VC-1; maximum resolution: 4K (4096 x 2160); maximum frame rate per second (FPS): 30</td></tr> System Memory</strong></td> 10 GB DDR3L</td></tr> Total Effective Storage</strong></td> 15.7TB</td></tr> Drives</strong></td> 2 x Seagate IronWolf Pro ST18000NT001 18TB 7200 RPM</td></tr> Network Interface</strong></td> 1GbE</td></tr> </tbody></table> NFS Config</h3> </th> </th></tr></thead> Maximum NFS Protocol</strong></td> NFSv4.1</td></tr> Read Packet Size</strong></td> 8KB</td></tr> Write Packet Size</strong></td> 8KB</td></tr> </tbody></table> SMB Config</h3> </th> </th></tr></thead> Maximum SMB protocol</strong></td> SMB3</td></tr> Minimum SMB protocol</strong></td> SMB1</td></tr> SMB range</strong></td> SMB1,5MB2,SMB2 and Large MTU,SMB3</td></tr> Transport encryption mode</strong></td> Client defined</td></tr> Enable server signing:</strong></td> Disable</td></tr> Enable Opportunistic Locking</strong></td> Enabled</td></tr> Enable SMB2 lease</strong></td> Disabled</td></tr> Enable SMB durable handles (Cross-protocol file locking will be disabled)</strong></td> Disabled</td></tr> </tbody></table> </th> </th></tr></thead> Enable Local Master Browser</strong></td> Disabled</td></tr> Enable DirSort VFS module</strong></td> Disabled</td></tr> Veto files</strong></td> Disabled</td></tr> Allow symbolic links within shared folders</strong></td> Enabled</td></tr> Allow symbolic links across shared folders</strong></td> Disabled</td></tr> *Disable multiple connections from the same IP address</td> Disabled</td></tr> Collect debug logs</strong></td> Disabled</td></tr> Apply default UNIX permissions</strong></td> Enabled</td></tr> Do not reserve disk space when creating files</strong></td> Disabled</td></tr> </tbody></table> Client</h2> A little confusing since in this case the Linux server is acting as a client to the NAS.</p> </th> </th></tr></thead> CPU Model</strong></td> AMD Ryzen™ 7 3750H</td></tr> CPU Architecture</strong></td> 64-bit</td></tr> CPU Frequency</strong></td> 4 Cores/8 Threads (4MB Cache , up to 4.0 GHz)</td></tr> System Memory</strong></td> 16 GB DDR4</td></tr> Network Interface</strong></td> 2.5GbE</td></tr> </tbody></table> Mount Configs</h3> The section of my cloud-init config that defines the NFS and SMB mounts.</p> #cloud-config </span> </span>mounts</span>: </span> - [ </span>"192.168.50.55:/volume1/backups"</span>, </span>"/mnt/nfs/ds218/backups"</span>, </span>"nfs4"</span>, </span>"rw,noatime,nodiratime,vers=4.1,soft,retrans=2,_netdev,async,rsize=32768,wsize=32768,fsc,cto,acl"</span>, </span>"0"</span>, </span>"0" </span>] </span> - [ </span>"//192.168.50.55/backups"</span>, </span>"/mnt/smb/ds218/backups"</span>, </span>"cifs"</span>, </span>"username=notmyusername,password=notmypassword,vers=3.1.1,uid=1000,_netdev"</span>, </span>"0"</span>, </span>"0" </span>] </span></code></pre> Network</h2> The NAS and server are connected wired through a 2.5GbE switch.</p> Methodology</h1> For more details on the exact script see fs_bench.sh</a>.</p> To test the filesystems speed, I ran the fs_bench.sh script which runs fio with the 5 test types on both mounts:</p> Sequential read</li> Sequential write</li> Random read</li> Random write</li> Mixed random read/write</li> </ul> Results</h1> For the raw results see fio_output.txt</a>. Otherwise here a summary created using the analysis.py</a> script follows in the next section.</p> Performance Comparison: nfs vs smb</h2> ❯ python3 analysis.py </span>Performance Comparison: NFS vs SMB </span>================================== </span> </span>SEQUENTIAL_READ: </span> IOPS: NFS: 110.00 SMB: 112.00 Difference: 1.82% </span> Bandwidth: NFS: 111.00 MiB/s SMB: 112.00 MiB/s Difference: 0.90% </span> Avg Latency: NFS: 36091.93 µs SMB: 35625.10 µs Difference: 1.31% </span> P99 Latency: NFS: 42206.00 µs SMB: 35914.00 µs Difference: 17.52% </span> CPU Usage: </span> User: NFS: 0.03% SMB: 0.04% Difference: 0.01% </span> System: NFS: 0.73% SMB: 0.46% Difference: -0.27% </span> Total: NFS: 0.76% SMB: 0.50% Difference: -0.26% </span> </span>SEQUENTIAL_WRITE: </span> IOPS: NFS: 107.00 SMB: 110.00 Difference: 2.80% </span> Bandwidth: NFS: 108.00 MiB/s SMB: 110.00 MiB/s Difference: 1.85% </span> Avg Latency: NFS: 37040.95 µs SMB: 36137.06 µs Difference: 2.50% </span> P99 Latency: NFS: 57410.00 µs SMB: 60556.00 µs Difference: -5.20% </span> CPU Usage: </span> User: NFS: 0.11% SMB: 0.13% Difference: 0.02% </span> System: NFS: 0.65% SMB: 1.23% Difference: 0.58% </span> Total: NFS: 0.76% SMB: 1.36% Difference: 0.60% </span> </span>RANDOM_READ: </span> IOPS: NFS: 7334.00 SMB: 5513.00 Difference: -24.83% </span> Bandwidth: NFS: 28.70 MiB/s SMB: 21.50 MiB/s Difference: -25.09% </span> Avg Latency: NFS: 541.17 µs SMB: 721.68 µs Difference: -25.01% </span> P99 Latency: NFS: 1319.00 µs SMB: 1352.00 µs Difference: -2.44% </span> CPU Usage: </span> User: NFS: 0.80% SMB: 1.05% Difference: 0.25% </span> System: NFS: 2.96% SMB: 2.56% Difference: -0.40% </span> Total: NFS: 3.76% SMB: 3.61% Difference: -0.15% </span> </span>RANDOM_WRITE: </span> IOPS: NFS: 6618.00 SMB: 7691.00 Difference: 16.21% </span> Bandwidth: NFS: 25.90 MiB/s SMB: 30.00 MiB/s Difference: 15.83% </span> Avg Latency: NFS: 599.69 µs SMB: 514.93 µs Difference: 16.46% </span> P99 Latency: NFS: 1532.00 µs SMB: 2008.00 µs Difference: -23.71% </span> CPU Usage: </span> User: NFS: 0.75% SMB: 1.33% Difference: 0.58% </span> System: NFS: 2.94% SMB: 10.33% Difference: 7.39% </span> Total: NFS: 3.69% SMB: 11.66% Difference: 7.97% </span> </span>MIXED_RANDRW: </span> IOPS: NFS: 3531.00 SMB: 3137.00 Difference: -11.16% </span> Bandwidth: NFS: 13.80 MiB/s SMB: 12.30 MiB/s Difference: -10.87% </span> Avg Latency: NFS: 588.18 µs SMB: 598.77 µs Difference: -1.77% </span> P99 Latency: NFS: 1516.00 µs SMB: 2376.00 µs Difference: -36.20% </span> CPU Usage: </span> User: NFS: 0.79% SMB: 1.17% Difference: 0.38% </span> System: NFS: 3.16% SMB: 5.37% Difference: 2.21% </span> Total: NFS: 3.95% SMB: 6.54% Difference: 2.59% </span> </span>Summary: </span> sequential_read: </span> Performance: SMB is faster by 1.82% </span> CPU Efficiency: SMB uses less CPU by 0.26% </span> sequential_write: </span> Performance: SMB is faster by 2.80% </span> CPU Efficiency: NFS uses less CPU by 0.60% </span> random_read: </span> Performance: NFS is faster by 33.03% </span> CPU Efficiency: SMB uses less CPU by 0.15% </span> random_write: </span> Performance: SMB is faster by 16.21% </span> CPU Efficiency: NFS uses less CPU by 7.97% </span> mixed_randrw: </span> Performance: NFS is faster by 12.56% </span> CPU Efficiency: NFS uses less CPU by 2.59% </span></code></pre> Conclusion</h1> The difference ended up being somewhat negligible for sequential reads and writes. With both NFS and SMB getting pretty close to the theoretical limit of 125MB/s of the 1GbE link.</p> The difference in random read was fairly large of about 30% in favor of NFS. While the random write speed was in favor of SMB with it being about 15% faster. And in a mixed workload of random reads and writes, NFS wins with the lightest of margins of about 12%. In these, it does seem like NFS is slightly more CPU efficient but not really by much.</p> Was all this worth it? Not really but at least now I know that id doesn't make much of a difference at least with my current setup. I have been planning on upgrading the NAS and network to 2.5GbE, I wonder if that would make a difference. At least now that I had an easy quick setup to test this, I can do it again once I upgrade.</p> My Personal Self-Hosted Infrastructure Part 1 Unknown — Mon, 09 Sep 2024 00:00:00 +0000 Introduction</h2> Over the past decade, there's been a noticeable shift in how software is delivered and consumed. Many applications that were once standalone programs are now served over the web. This trend towards web-based services and software-as-a-service (SaaS) models has its merits, but it also opens up opportunities for individuals to take control of their digital lives through self-hosting. In this post, I'll explore the reasons for this shift, the benefits of self-hosting, and share my personal journey into building a self-hosted infrastructure.</p> Why Web Services?</h2> The move towards web-based services has several advantages:</p> Personal Benefits:</p> Access to apps and data from anywhere</li> Easier collaboration on shared projects</li> Simple sharing of content like photo galleries or works-in-progress</li> </ul> Technical Benefits:</p> Access to software through APIs</li> Ability to schedule tasks remotely</li> Offloading processing to potentially more powerful servers</li> </ul> While these benefits could theoretically be achieved through desktop applications or command-line interfaces, web services with HTTP JSON APIs provide one of the simplest ways to reap these advantages.</p> The Case for Self-Hosting</h2> Despite the convenience of services like Google Workspace or Dropbox, there are compelling reasons to consider self-hosting:</p> Data Control: You maintain ownership and control over your data.</li> Customization: You can modify and adapt services to your specific needs.</li> Privacy: Your data remains on your own infrastructure.</li> Learning Opportunity: Self-hosting provides hands-on experience with various technologies.</li> </ol> Self-hosting does come with its challenges, but starting small and focusing on useful, frequently-used services can help you build a sustainable personal infrastructure.</p> How It Started: My Self-Hosting Journey</h2> My venture into self-hosting began in November 2017 with the purchase of a Synology DS218+ NAS, equipped with an 8TB WD Red Pro drive. Initially intended for expanded storage and photo backups, I soon discovered its potential as a home server through Docker support.</p> The first services I installed were:</p> Plex: For media organization and playback</li> Transmission: A torrent client</li> </ol> These services, which I had previously run locally, now benefited from operating on a dedicated, always-on server.</p> I started by using the Docker UI provided by Synology, which was sufficient for managing these two services. For remote access, I utilized Synology's DDNS service.</p> As I expanded my setup, I learned valuable lessons about storage management and hardware selection:</p> Understanding RAID: When my original 8TB drive began to fill up, I realized I needed to choose between additional storage (RAID 0) or data redundancy (RAID 1). I opted for RAID 1 to protect my important photos and videos.</li> Scaling Up: To increase storage capacity, I eventually purchased a larger NAS, the Synology DS1019+ with 5 bays. This experience taught me the importance of anticipating future needs when selecting hardware.</li> Start Small, Think Big: If I could advise my past self, I'd recommend thoroughly evaluating needs before investing in hardware. The adage "buy once, cry once" applies well to NAS and server setups. For those unsure about their requirements, starting with a used mini PC can be an excellent way to test use cases before committing to more expensive hardware.</li> </ol> Conclusion</h2> Self-hosting offers a powerful way to take control of your digital life, but it requires careful planning and a willingness to learn. As my journey shows, starting small and gradually expanding can lead to a robust and personalized infrastructure that meets your specific needs.</p> Technical Terms Glossary</h2> RAID (Redundant Array of Independent Disks)</strong>: A storage technology that combines multiple disk drive components into a logical unit for data redundancy and performance improvement. Common RAID levels include: RAID 0: Stripes data across drives for improved performance but no redundancy.</li> RAID 1: Mirrors data on two drives for full redundancy.</li> </ul> </li> DDNS (Dynamic DNS)</strong>: A method of automatically updating a name server in the Domain Name System with the active DHCP address. This is particularly useful for accessing home servers with dynamic IP addresses.</li> NAS (Network Attached Storage)</strong>: A file-level computer data storage server connected to a computer network providing data access to heterogeneous network clients.</li> Docker</strong>: A platform for developing, shipping, and running applications in containers, which are lightweight, standalone, executable packages of software that include everything needed to run an application.</li> </ul> RFC Processes for Software Development Unknown — Wed, 14 Aug 2024 00:00:00 +0000 Documenting at a high level what you want to achieve before committing to the work is often a good idea. Especially for complex projects or ideas. This allows others to look over the proposed changes, poke holes at it or be made aware of changes that could affect them.</p> A few ways I have seen this done at different companies are Technical Designs (TDs), Architecture Decision Records (ADRs) and Request for Comments.</p> I have found that a great way to write these kind of documents is in a format not too dissimilar to IETF RFCs. Those often have the right level of detail and a structure that lends itself well to software ideas. Words such as must, shall, should, etc as defined in RFC-2119</a> are also useful to communicate the right level of requirement.</p> For documenting decisions at a project level, such as a tool or individual service, ADRs can be useful where a change can affect the overall project or projects interacting with it. I found that ADRs often break down and don't work well when multiple teams and projects are in one Architecture Decision Log (ADL, a collection of ADRs). They work better in high context situations where ADRs can be short and teams have enough understanding of the project to not have too much background details in ADRs.</p> Depending on your company or situation, you might want to make these documents more or less formal and structured.</p> One good example of an RFC process used at a private company for which their documents are public is Oxide Computer with their RFD documents. Requests for Discussion</a>.</p> Trying to understand how Next.js render server components on updates Unknown — Fri, 31 May 2024 00:00:00 +0000 I was reading the Next.js App Router course</a>, mostly for my own front end curiosity but also because I was looking at doing some frontend projects, as one does when you are frontend curious but mostly do backend development. When I got to Chapter 11: Adding Search and Pagination</a>, I was having a hard time really understanding what was happening and how Next was rendering the updates to the page.</p> First let's start with what this part of the Next.js tutorial is about and how it explains it.</p> You start with a fairly static page.</p> import </span>Pagination </span>from </span>'@/app/ui/invoices/pagination'</span>; </span>import </span>Search </span>from </span>'@/app/ui/search'</span>; </span>import </span>Table </span>from </span>'@/app/ui/invoices/table'</span>; </span>import </span>{ CreateInvoice } </span>from </span>'@/app/ui/invoices/buttons'</span>; </span>import </span>{ lusitana } </span>from </span>'@/app/ui/fonts'</span>; </span>import </span>{ InvoicesTableSkeleton } </span>from </span>'@/app/ui/skeletons'</span>; </span>import </span>{ Suspense } </span>from </span>'react'</span>; </span> </span>export default async </span>function </span>Page</span>() { </span> </span>return </span>( </span> <</span>div </span>className</span>=</span>"w-full"</span>> </span> <</span>div </span>className</span>=</span>"flex w-full items-center justify-between"</span>> </span> <</span>h1 </span>className</span>=</span>{</span>`${lusitana.className} text-2xl`</span>}>Invoices</</span>h1</span>> </span> </</span>div</span>> </span> <</span>div </span>className</span>=</span>"mt-4 flex items-center justify-between gap-2 md:mt-8"</span>> </span> <</span>Search </span>placeholder</span>=</span>"Search invoices..." </span>/> </span> <</span>CreateInvoice </span>/> </span> </</span>div</span>> </span> {</span>/ <Suspense key={query + currentPage} fallback={<InvoicesTableSkeleton />}> </span> <Table query={query} currentPage={currentPage} /> </span> </Suspense> /</span>} </span> <</span>div </span>className</span>=</span>"mt-5 flex w-full justify-center"</span>> </span> {</span>/ <Pagination totalPages={totalPages} /> /</span>} </span> </</span>div</span>> </span> </</span>div</span>> </span> ); </span>} </span></code></pre> The 2 main components we care about are the <Search></code> and <Table></code> components. The <Search></code> component is the input that will take a query and somehow update the results in the <Table></code>. The tutorial explains that the best way to handle this is through using the page's url query params with useSearchParams</code>.</p> The Search</code> component then looks like this:</p> 'use client'</span>; </span> </span>import </span>{ MagnifyingGlassIcon } </span>from </span>'@heroicons/react/24/outline'</span>; </span>import </span>{ useSearchParams, usePathname, useRouter } </span>from </span>'next/navigation'</span>; </span> </span>export default </span>function </span>Search</span>() { </span> </span>const </span>searchParams </span>= </span>useSearchParams</span>(); </span> </span>const </span>pathname </span>= </span>usePathname</span>(); </span> </span>const </span>{ replace } </span>= </span>useRouter</span>(); </span> </span> </span>function </span>handleSearch</span>(</span>term</span>: </span>string</span>) { </span> </span>const </span>params </span>= new </span>URLSearchParams(searchParams); </span> </span>if </span>(term) { </span> params.</span>set</span>(</span>'query'</span>, term); </span> } </span>else </span>{ </span> params.</span>delete</span>(</span>'query'</span>); </span> } </span> </span>replace</span>(</span>`${pathname}?${params.</span>toString</span>()}`</span>); </span> } </span>//... </span>} </span></code></pre> Note that part of the component was elided. See the full component in the Git repo</a>.</p> Next, we need to wire the query params that were updated by our Search</code> component into the <Table></code> component.</p> The page will now look like this:</p> import </span>Pagination </span>from </span>'@/app/ui/invoices/pagination'</span>; </span>import </span>Search </span>from </span>'@/app/ui/search'</span>; </span>import </span>Table </span>from </span>'@/app/ui/invoices/table'</span>; </span>import </span>{ CreateInvoice } </span>from </span>'@/app/ui/invoices/buttons'</span>; </span>import </span>{ lusitana } </span>from </span>'@/app/ui/fonts'</span>; </span>import </span>{ Suspense } </span>from </span>'react'</span>; </span>import </span>{ InvoicesTableSkeleton } </span>from </span>'@/app/ui/skeletons'</span>; </span> </span>export default async </span>function </span>Page</span>({ </span> </span>searchParams</span>, </span>}</span>: </span>{ </span> searchParams</span>?: </span>{ </span> query</span>?: </span>string</span>; </span> page</span>?: </span>string</span>; </span> }; </span>}) { </span> </span>const </span>query </span>= </span>searchParams?.query </span>\|\| </span>''</span>; </span> </span>const </span>currentPage </span>= </span>Number</span>(searchParams?.page) </span>\|\| </span>1</span>; </span> </span> </span>return </span>( </span> <</span>div </span>className</span>=</span>"w-full"</span>> </span> <</span>div </span>className</span>=</span>"flex w-full items-center justify-between"</span>> </span> <</span>h1 </span>className</span>=</span>{</span>`${lusitana.className} text-2xl`</span>}>Invoices</</span>h1</span>> </span> </</span>div</span>> </span> <</span>div </span>className</span>=</span>"mt-4 flex items-center justify-between gap-2 md:mt-8"</span>> </span> <</span>Search </span>placeholder</span>=</span>"Search invoices..." </span>/> </span> <</span>CreateInvoice </span>/> </span> </</span>div</span>> </span> <</span>Suspense </span>key</span>=</span>{query </span>+ </span>currentPage} </span>fallback</span>=</span>{<</span>InvoicesTableSkeleton </span>/>}> </span> <</span>Table </span>query</span>=</span>{query} </span>currentPage</span>=</span>{currentPage} /> </span> </</span>Suspense</span>> </span> <</span>div </span>className</span>=</span>"mt-5 flex w-full justify-center"</span>> </span> {</span>/ <Pagination totalPages={totalPages} /> /</span>} </span> </</span>div</span>> </span> </</span>div</span>> </span> ); </span>} </span></code></pre> Now, whenever the <Search></code> component updates the query params with the search query entered, the <Table></code> will have the search query passed into it's props so that it can show the right results.</p> Finally, let's take a look at what's happening in the <Table></code> component.</p> // ... </span>export default async </span>function </span>InvoicesTable</span>({ </span> </span>query</span>, </span> </span>currentPage</span>, </span>}</span>: </span>{ </span> query</span>: </span>string</span>; </span> currentPage</span>: </span>number</span>; </span>}) { </span> </span>const </span>invoices </span>= await </span>fetchFilteredInvoices</span>(query, currentPage); </span> </span>// ... </span>} </span></code></pre> This shows that the <Table></code> component calls an async function fetchFilteredInvoices</code> with the query and currentPage obtained from the Props of the component. This async function could actually fetch the invoices from any data source. In the Next.js tutorial, it's from a database covered in a previous chapter.</p> The tutorial for this chapter actually continues and covers a few more things: input debouncing (which which delays triggering any change until what is triggering the change stops, in this case it's the user typing) and pagination. But for our purposes, we can stop here and summarize what we and the tutorial covers and see we can understand what happened.</p> Quick Summary</h1> Our page has 2 components, a <Search></code> and a <Table></code> component. These 2 components interact with each other through the url query parameters of the page.</p> The <Search></code> component takes user input of the search query and updates the query parameter of the page with that search query. e.g. the user enters Lee</code> the url changes to /dashboard/invoices?query=Lee</code>.</p> The <Table></code> component takes the query parameters as it's Props and renders the right data.</p> When the user types a new query into the search box, the <Table></code> component updates the data.</p> But how is this all happening? Does the page refresh when the query parameters are updated? How does the table receive the updated data based on the search terms?</p> Maybe I am particularly thick or I might have missed something while reading this tutorial sleepy right before bed. But I had all these questions popping up in my head while reading this Chapter.</p> Chapter 9: Streaming</a> does discuss how you can chunks of a page by loading partial content initially and then streaming in the rest of the content is chunks. This uses React <Suspense></code> feature or the Next.js loading.tsx</code> feature. Maybe Next.js is streaming in the data to the component?</p> Chapter 10: Partial Prerendering</a> covers how a page can be both static and dynamic. It describes the feature at a high level but I am not entirely sure if this is related to what I am missing.</p> Let's see how it works</h1> I have often found that the best way to understand how something works is to build and poke at what you want to understand. In this case, since this is a web thing, we can easily poke at it using dev tools. So let's do it.</p> We can reuse most of the code from the tutorial, except for the fetchFilteredInvoices</code> function which uses a database and was setup in Chapter 6 and 7</a>. Instead we will mock this and use the data provided in the tutorial to populate the database.</p> import </span>{ customers, invoices } </span>from </span>'@/app/lib/placeholder-data'</span>; </span> </span>const </span>FETCH_SIMULATED_DELAY </span>= </span>3000</span>; </span>// delay in milliseconds </span> </span>export async </span>function </span>fetchFilteredInvoices</span>(</span>query</span>: </span>string</span>, </span>currentPage</span>: </span>number</span>)</span>: </span>Promise<{ </span> name</span>: </span>string </span>\| </span>undefined</span>, </span> email</span>: </span>string </span>\| </span>undefined</span>, </span> image_url</span>: </span>string </span>\| </span>undefined</span>, </span> customer_id</span>: </span>string</span>, </span> amount</span>: </span>number</span>, </span> status</span>: </span>string</span>, </span> date</span>: </span>string </span>}[]> { </span> </span>return new </span>Promise((</span>resolve</span>) </span>=> </span>{ </span> </span>setTimeout</span>(() </span>=> </span>{ </span> </span>// Combine invoices with customer details </span> </span>const </span>detailedInvoices </span>= </span>invoices.</span>map</span>(</span>invoice </span>=> </span>{ </span> </span>const </span>customer </span>= </span>customers.</span>find</span>(</span>c </span>=> </span>c.id </span>=== </span>invoice.customer_id); </span> </span>return </span>{ </span> </span>...</span>invoice, </span> name: customer?.name, </span> email: customer?.email, </span> image_url: customer?.image_url, </span> }; </span> }); </span> </span> </span>// Simulate filtering based on the query </span> </span>const </span>filteredInvoices </span>= </span>detailedInvoices.</span>filter</span>(</span>invoice </span>=> </span> invoice.name?.</span>toLowerCase</span>().</span>includes</span>(query.</span>toLowerCase</span>()) </span> ); </span> </span> </span>// Simulate pagination </span> </span>const </span>pageSize </span>= </span>10</span>; </span> </span>const </span>startIndex </span>= </span>(currentPage </span>- </span>1</span>) </span> </span>pageSize; </span> </span>const </span>paginatedInvoices </span>= </span>filteredInvoices.</span>slice</span>(startIndex, startIndex </span>+ </span>pageSize); </span> </span> </span>resolve</span>(paginatedInvoices); </span> }, FETCH_SIMULATED_DELAY); </span>// Simulate an x second delay </span> }); </span>} </span></code></pre> For good measure, I also added a delay to the fetchFilteredInvoices</code> function to simulate calling an external data source.</p> Let's run this example and see what's happening.</p> ></span> npm run dev </span></code></pre> ![first_page_load] Insert screenshot here</p> On first page load everything is as expected. The initial request is a call to /dashboard/invoices</code> which gets an html response. This initial request is followed by several various js requests to load the script to bootstrap the client side application.</p> But what happens when we enter a search query?</p> ![search_query_partial_render] Insert screenshot</p> We can see that the application does 2 requests. We can finally see what they are doing by inspecting them. What we see are that the requests on /dashboard/invoices</code> with specific headers and query parameters, return a react tree response. This react tree is the part of the page that were updated. The first request is the initial render of the Skeleton Table in the <Suspense></code> tags while the fetchFilteredInvoices</code> loaded the data to be rendered. The second request is then the <Table></code> component rendered with the loaded data.</p> // 1. First Request </span>await </span>fetch</span>(</span>"http://localhost:3000/dashboard/invoices?query=Lee&_rsc=14zw6"</span>, { </span> </span>"credentials"</span>: </span>"include"</span>, </span> </span>"headers"</span>: { </span> </span>"User-Agent"</span>: </span>"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"</span>, </span> </span>"Accept"</span>: </span>"/"</span>, </span> </span>"Accept-Language"</span>: </span>"en-US,en;q=0.5"</span>, </span> </span>"RSC"</span>: </span>"1"</span>, </span> </span>"Next-Router-State-Tree"</span>: </span>"%5B%22%22%2C%7B%22children%22%3A%5B%22dashboard%22%2C%7B%22children%22%3A%5B%22invoices%22%2C%7B%22children%22%3A%5B%22__PAGE__%22%2C%7B%7D%5D%7D%5D%7D%5D%7D%2Cnull%2Cnull%2Ctrue%5D"</span>, </span> </span>"Next-Router-Prefetch"</span>: </span>"1"</span>, </span> </span>"Next-Url"</span>: </span>"/dashboard/invoices"</span>, </span> </span>"Sec-Fetch-Dest"</span>: </span>"empty"</span>, </span> </span>"Sec-Fetch-Mode"</span>: </span>"cors"</span>, </span> </span>"Sec-Fetch-Site"</span>: </span>"same-origin"</span>, </span> </span>"Priority"</span>: </span>"u=4" </span> }, </span> </span>"referrer"</span>: </span>"http://localhost:3000/dashboard/invoices"</span>, </span> </span>"method"</span>: </span>"GET"</span>, </span> </span>"mode"</span>: </span>"cors" </span>}); </span> </span>// 2. Second Request </span>await </span>fetch</span>(</span>"http://localhost:3000/dashboard/invoices?query=Lee&_rsc=181do"</span>, { </span> </span>"credentials"</span>: </span>"include"</span>, </span> </span>"headers"</span>: { </span> </span>"User-Agent"</span>: </span>"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"</span>, </span> </span>"Accept"</span>: </span>"/"</span>, </span> </span>"Accept-Language"</span>: </span>"en-US,en;q=0.5"</span>, </span> </span>"RSC"</span>: </span>"1"</span>, </span> </span>"Next-Router-State-Tree"</span>: </span>"%5B%22%22%2C%7B%22children%22%3A%5B%22dashboard%22%2C%7B%22children%22%3A%5B%22invoices%22%2C%7B%22children%22%3A%5B%22__PAGE__%3F%7B%5C%22query%5C%22%3A%5C%22Lee%5C%22%7D%22%2C%7B%7D%2Cnull%2C%22refetch%22%5D%7D%5D%7D%5D%7D%5D"</span>, </span> </span>"Next-Url"</span>: </span>"/dashboard/invoices"</span>, </span> </span>"Sec-Fetch-Dest"</span>: </span>"empty"</span>, </span> </span>"Sec-Fetch-Mode"</span>: </span>"cors"</span>, </span> </span>"Sec-Fetch-Site"</span>: </span>"same-origin"</span>, </span> </span>"Priority"</span>: </span>"u=4" </span> }, </span> </span>"referrer"</span>: </span>"http://localhost:3000/dashboard/invoices"</span>, </span> </span>"method"</span>: </span>"GET"</span>, </span> </span>"mode"</span>: </span>"cors" </span>}); </span></code></pre> 0:["development",[["children","dashboard","children","invoices","children","__PAGE__?{\"query\":\"Lee\"}",["__PAGE__?{\"query\":\"Lee\"}",{}],null,null]]] </span></code></pre> 0:["development",[["children","dashboard","children","invoices","children","__PAGE__?{\"query\":\"Lee\"}",["__PAGE__?{\"query\":\"Lee\"}",{}],"$L1",[null,"$L2"]]]] </span>1:["$L3","$L4",null] </span>2:[["$","meta","0",{"name":"viewport","content":"width=device-width, initial-scale=1"}],["$","meta","1",{"charSet":"utf-8"}],["$","meta","2",{"name":"next-size-adjust"}]] </span>5:I["(app-pages-browser)/./app/ui/search.tsx",["app/dashboard/invoices/page","static/chunks/app/dashboard/invoices/page.js"],""] </span>6:"$Sreact.suspense" </span>4:["$","div",null,{"className":"w-full","children":[["$","div",null,{"className":"flex w-full items-center justify-between","children":["$","h1",null,{"className":"__className_712214 text-2xl","children":"Invoices"}]}],["$","div",null,{"className":"mt-4 flex items-center justify-between gap-2 md:mt-8","children":["$","$L5",null,{"placeholder":"Search invoices..."}]}],["$","$6","Lee1",{"fallback":["$","div",null,{"className":"mt-6 flow-root","children":["$","div",null,{"className":"inline-block min-w-full align-middle","children":["$","div",null,{"className":"rounded-lg bg-gray-50 p-2 md:pt-0","children":[["$","div",null,{"className":"md:hidden","children":[["$","div",null,{"className":"mb-2 w-full rounded-md bg-white p-4","children":[["$","div",null,{"className":"flex items-center justify-between border-b border-gray-100 pb-8","children":[["$","div",null,{"className":"flex items- </span>... </span></code></pre> Summary</h1> So we were able to answer some of the questions we had initially. Next.js manages the request lifecycle of the application. It does a full page load initially where it is necessary, but once loaded, replaces only parts of the page that are necessary. No full page reload happens when the query params are updated on the page or when new data is rendered to the page. The data is passed through the page's url query parameters mediated through next/navigation</code> which manages the updates to the page when necessary.</p> sequenceDiagram </span> User ->> UserAgent: Load /dashboard/invoices </span> UserAgent ->> Next.js Server: Request /dashboard/invoices </span> activate Next.js Server </span> Next.js Server -->> UserAgent: text/html </span> deactivate Next.js Server </span> </span> User ->> UserAgent: Enters search query </span> UserAgent ->> Next.js Server: Request /dashboard/invoices?query=...&_rsc=... </span> activate Next.js Server </span> Note right of Next.js Server: Next.js renders the changes </span> Next.js Server -->> UserAgent: text/x-component of the changes </span> deactivate Next.js Server </span></code></pre> Check out the repo where I tried to create a minimal example of what I looked into here. Github link</a></p> Playing with Automerge and text CRDTs Unknown — Sat, 01 Feb 2020 00:00:00 +0000 You will find the code and output from the work described here at the following links:</p> Github Repo</a></li> Web App</a></li> </ul> Automerge: a Javascript CRDT library</h2> I had heard of Conflict-free replicated data types</a> before but their use crystalized in my mind recently when I found out about the Automerge</a> Javascript library. It's blurb states that it's a JSON-like data structure that can be modified concurrently and merged automatically. What this means is that it enables the creation of collaborative applications where multiple users could modify the same structure locally, send their changes over the network and have their changes merge together even when they made conflicting modifications. It is of course not perfect since the library cannot know in advance what the right state would be if conflicting modifications happened on the same property if it doesn't actually support it, in these cases, it chooses one of the updates and saves the other in it's conflicts. What's great about it's conflict resolution, is that when it chooses one resolution, all the nodes using automerge receiving the conflicting changes will also resolve it in the same consistent way.</p> One of the data structures I was interested on was the Text</code> data type, which promises to enable text editors to become collaborative and allow concurrent updates even offline. With that in mind, a lot of use cases started to pop into my mind, since I have been thinking about a better way of storing my notes recently. It matches several criteria that could help be useful for my use case, namely, being in Javascript, I could use it to build a web app to sync my notes in several places and even enable merging conflicting updates when I come back online from different devices.</p> But I haven't been working much with frontend web technologies recently, so I decided to start small and experiment with the library with a small prototype. This is what I learned while doing that.</p> React and Redux</h2> Any good frontend project has to start with some kind of rendering library or framework. I could go vanilla js, or choose one of the many frontend libraries available today. One that seems to have largely taken over the frontend world these days seem to be React. I used React at a startup several years ago and from a cursory look, the ecosystem seems to have moved forward quite well, the APIs are largely the same, but the breath of libraries, tooling and documentation has improved a lot in the intervening years. So I decided to go with a React/Redux rendering stack.</p> This architecture is quite similar to Elm</a>, which I have been using in some of my side projects. I should probably write about it sometime, but today isn't about Elm. Let's just say that Elm is a functional programming language for frontend development with a strong focus on no runtime errors (at least not ones of the typed kind). Since it's the tech I have been using the most in my side projects where I need a web frontend, that is the starting point for my comparisons.</p> The first thing I noticed digging around is how much more breath of libraries and integrations there are for a React application. By comparison in Elm, I found myself feeling a little isolated by lack of libraries or different tooling from the rest of the web ecosystem.</p> I also found the improved documentation around React since the last time I used it to be very nice. The React, Redux and Create React App official websites had a lot of examples and pragmatic use cases to help.</p> The onboarding process for a new project seems to have been streamlined greatly since I used React. The Create React App and react-scripts enables you to have a working project from a template and configuration less build pretty quick. You can of course configure your own bundler but I enjoy being able to start on my project fairly quick and only needing to configure the bundler once I have a specific need to do that.</p> Typescript</h2> I am not big fan of Javascript, it's dynamic type system, it's prototype base inheritance or it's sometimes confusing scoping rules and it's many other faults don't help when I don't use the language often. So seeing typescript coming up in popularity in the Javascript ecosystem fills me with much joy. Even if Typescript doesn't have the strictest type system or one that covers all cases that can occur in Javascript, it's still better than nothing. That is why I wanted to get some familiarity with it.</p> After about 15mins on its documentation</a>, I found myself having enough of a grasp of the language to start using it in a project. So that's encouraging. React</a>, Redux</a> and create-react-app</a> all have some documentation for using them with typescript, which is also a huge help to get started. I also found that most libraries that I encountered had typescript types either included with the library or as a @type/</code> package.</p> The only issue I encountered was went I tried to use immer</a> for my redux reducers. It cause strange type errors causing the Automerge objects stored in my redux store to not be typed as Automerge types because of the readonly properties added by immer it seemed. I didn't really debug much further and quickly reverted back to plain reducers. If I were to use it in the future, I would try to debug the issue, but that does give me pause about adding libraries and their interactions between each other and typescript.</p> Issues with Webpack prod build</h2> The final problem I encountered was at the very end of my experiment. During my whole development, I had been using it's development mode to run my app and it was working fine until I decided to run npm run build</code>. The production build started to give me undefined errors on a class method I was calling on an Automerge object. Looking strictly at the code was no help since from the typescript source code, I was calling the method correctly to bind this</code> and it was working correctly on the development build. It was only after asking for some help from a coworker with much more experience on the front end that we ended up debugging the issue to the bundled and minified code from webpack. It required disabling source maps and locating the corresponding code in the transpiled output using judiciously placed unique literal strings that we found that the code wasn't being transpiled to the same output we would expect. The solution we ended up implementing was to add a .bind()</code> which looked something like this.</p> src/utils/automerge.js</div> ... </span>doc.text.insertAt?.</span>bind</span>(doc.text)(idx, changedText); </span>... </span></code></pre> This did the trick but doesn't inspire me with a ton of confidence in the tooling in the Javascript ecosystem.</p> Conclusions</h2> This has been a bit of an off the cuff post on my experience getting back into frontend development and my experiment with Typescript and the Automerge CRDT library. Overall, I found that the Javascript ecosystem has a huge breath of libraries to do almost anything. That Typescript is pretty awesome but the integration with the libraries you want to use might not be perfect. Bringing libraries to your project can have lots of risks and you should only bring them in if you feel confident in truly owning those dependencies, and that's especially true in an ecosystem with so many libraries.</p> Finally, the goal of the experiment to see if Automerge was a viable data structure for a generic note editor, indicated to me that although it works amazingly well, it may not be the best match for the potential generic note editor project I had in mind. The updates to the Text data structure took a bit too long (moderate updates to the text sometimes took over 4s) and the storage overhead of the data structure was also way too high (3KB of text took 650KB of storage for the data structure which is about a 200x overhead).</p> Check out the code in the Github repo</a> and the web app</a>.</p> I will keep an eye out for it in the future as it improves. The author is working on improving the performance of the Text data structure and overall library as these issue indicate.</p> https://github.com/automerge/automerge/issues/212</li> https://github.com/automerge/automerge/issues/89</li> </ul> Integrating Ory Hydra OAuth 2.0 with a Rust web service Unknown — Sat, 18 Jan 2020 00:00:00 +0000 I have been using Rust quite a bit more over the last year in a variety of use cases and I have been quite satisfied with the language, libraries and the community. So I think it might be time to write a little bit about how well suited Rust works in one area that I have worked a lot in, and that is HTTP APIs.</p> For this post I will be covering using Rust to write a web service that integrates with the Ory Hydra OAuth 2.0 Server and OpenID Connect Provider</a>. I think that might give a decent idea of the features offered by Rust and Rust libraries for web development.</p> Before we start, all the code we will be working on will be pushed to the following repository</a></p> What will we be doing in this post:</p> What is OAuth 2.0?</a> When should you used it?</a></li> Implementing authorization with OAuth 2.0</a></li> </ul> </li> Hydra: An OpenID Certified OAuth 2.0 Server and OpenID Connect Provider</a> Integrating with Hydra</a></li> </ul> </li> A Rust web service</a> Hello World</a></li> Login and consent</a></li> Setting up the Hydra services</a></li> Generating the Hydra OpenApi Client</a></li> Updating our login and consent implementation</a> Enabling logging</a></li> </ul> </li> Testing our login and consent integration</a></li> Async Rust</a></li> </ul> </li> Where to go from here?</a></li> References</a></li> </ul> What is OAuth 2.0?</h2> The OAuth 2.0 authorization framework enables a third-party application to obtain limited access to an HTTP service, either on behalf of a resource owner by orchestrating an approval interaction between the resource owner and the HTTP service, or by allowing the third-party application to obtain access on its own behalf. From RFC 6749: The OAuth 2.0 Authorization Framework</a></p> </blockquote> Essentially OAuth 2.0 enables applications (servers, mobile apps, SPAs, etc) to obtain access tokens to access an HTTP service scoped only to authorized actions/resources without needing the username and password of the account it would like to obtain access on behalf of.</p> As an almost too short to be correct summary of the OAuth 2.0 Authorization Framework, it defines 2 endpoints, the authorization and the token endpoints. The first is to request authorization with the scopes one would like to receive and the second is to request tokens. It also defines a set of grant types (authorization code, client credentials, etc...) and flows to follow for the different grant types. As for crypto, the protocol simply defines that it needs to be over https. So what we will be doing here won't actually be to spec until we put TLS termination into our system.</p> There are also several extensions to the protocol, the most important one we will be interested in in this post is the Introspection endpoint</a> which defines how resource servers communicate with the authorization server to verify access tokens. Another important one that might be of interested but that we won't cover in this post is the OpenID Connect 1.0 specification</a> which is a protocol build over OAuth 2.0.</p> An example use case where we would use OAuth 2 is when you would like to give access to your youtube account to an application to manage your comments. You wouldn't want that application to also be able to post videos for you or change your profile page. You can do this with OAuth 2.0 by authorizing that application if it has an OAuth 2.0 client with Youtube. That application can go through an OAuth 2.0 flow that will give it an access token that it can then use to get access to manage your comments.</p> Although at first glance, the protocol may sound simple, it is anything but, implementing a spec compliant and interoperable version of the protocol that doesn't have security holes, takes some serious work and no one should trust such an implementation without at least an audit. So how can we use it then if we don't have deep pockets and all the time in the world to write one? That's what we will try to answer in this post using an existing and well tested OAuth 2.0 server with a well documented API.</p> When should you used it?</h3> Given the additional complexity that implementing and integration an OAuth 2.0 server adds to a project, why should you use OAuth 2?</p> The rule of thumb for when you need an OAuth 2.0 implementation, is if you need the ability to authorize third party applications to access your services. Otherwise, if you only need the ability to authorize users on your services through first party clients or applications, an encrypted cookie may better serve you.</p> For more information, check out the following post describing different types of access control and authorization patterns</a>.</p> Implementing authorization with OAuth 2.0</h3> Although it is of course possible to implement an OAuth 2.0 server from scratch, it is strongly not recommended to do so. Getting something wrong with the fundamental building blocks of the security of your system is not to be taken lightly. Another way to implement an OAuth 2 server is with your language's trusted OAuth 2 libraries. Being maintained and kept updated by domain experts, means that your service will have more eyes to find bugs as more people would be using them and gives your an OAuth implementation almost for free.</p> But after having implemented and maintained OAuth2 servers with the Spring OAuth 2 library in Java, I have grown a strong distaste towards this solution too. The reasons for that could be a post in and of itself, but most of the issues stem from maintenance of such such servers when they also contain much of the business logic for the access and authorization rules. It doesn't mean that this needs to be the case but it is often the case in my experience when we go this way.</p> That is why my preference nowadays lean towards the 3rd solution to implementing an authorization server, by using an external blessed implementation with good integration mechanisms. This forces you to keep your business logic and custom authorization rules seperate from a pure spec OAuth 2.0 or OpenID Connect server.</p> Hydra: An OpenID Certified OAuth 2.0 Server and OpenID Connect Provider</h2> This is where Hydra</a> comes in, a hardened, OpenID Certified OAuth 2.0 Server and OpenID Connect Provider optimized for low-latency, high throughput, and low resource consumption. Instead of implementing an OAuth2 server yourself, having to verify your implementation complies with the spec and doesn't have any security holes, we can use Hydra to provide the OAuth2 server functionality and implement the authentication and authorization logic as separate services.</p> Integrating with Hydra</h3> To integrate with the Hydra OAuth2 server, we need to implement a login and consent application. You can review the documentation for doing so here</a>.</p> A Rust web service</h2> If you have never programmed in rust, I would suggest The Rust Book</a> as the introduction to the language.</p> First lets start with a rust web service. Let's create a rust project.</p> cargo new hydra-auth-example-rs </span></code></pre> Now inside of the newly created project directory called hydra-auth-example-rs</code>, you should find 2 things, a src directory with a main.rs</code> file and a Cargo.toml</code> file.</p> $ tree </span>. </span>├── Cargo.toml </span>└── src </span> └── main.rs </span> </span>1 directory, 2 files </span></code></pre> To do anything we will need some dependencies to get our server working. Add the following below in [dependencies]</code>.</p> cargo.toml</div> ...</span> </span>[dependencies] </span>warp </span>= { </span>git </span>= </span>"https://github.com/seanmonstar/warp.git"</span>, </span>rev </span>= </span>"e94309e274872efed34e2a80f1e4553a45963510" </span>} </span>tokio </span>= </span>"0.2.6" </span> </span>log </span>= </span>"0.4.8" </span>env_logger </span>= </span>"0.7" </span></code></pre> Hello World</h3> We can now use these dependencies to provide a basic web service. Let's write a basic hello world endpoint as in the example.</p> src/main.rs</div> use </span>warp::{self, path, Filter}; </span> </span>fn </span>main</span>() { </span> </span>// GET /hello/warp => 200 OK with body "Hello, warp!" </span> </span>let</span> hello </span>= </span>path!(</span>"hello" </span>/ </span>String</span>).</span>map</span>(\|</span>name</span>\| format!(</span>"Hello, </span>{}</span>!"</span>, name)); </span> </span> warp::serve(hello).</span>run</span>(([</span>127</span>, </span>0</span>, </span>0</span>, </span>1</span>], </span>3000</span>)); </span>} </span></code></pre> We can run this hello world service with the following command in the project directory.</p> cargo run </span></code></pre> Or to create a release build.</p> cargo build --release </span># and to run this we will find the built binary at </span>./target/release/hydra-auth-example-rs </span></code></pre> To test this endpoint, you can call it with curl.</p> $ curl http://localhost:3000/hello/jon </span>Hello, jon!% </span></code></pre> Since we will also need login and consent pages for the Hydra integration, let's test html templating. For that we will be using the Tera templating library</a>. There are many other templating libraries in rust and you can find a non-exhaustive list here</a>. So let's add that dependency and use it.</p> Cargo.toml</div> ...</span> </span>[dependencies] </span>...</span> </span> </span>tera </span>= </span>"1.0.1" </span></code></pre> src/main.rs</div> use </span>tera::{Context, Tera}; </span>use </span>warp::{self, path, Filter}; </span> </span>#[tokio::main] </span>async </span>fn </span>main</span>() { </span> </span>let </span>mut</span> tera </span>= </span>Tera::default(); </span> tera.</span>add_raw_template</span>( </span> </span>"hello.html"</span>, </span> </span>r</span>#" </span><!DOCTYPE html> </span><head> </span> <meta charset="UTF-8"> </span> <title>Hello</title> </span></head> </span><body> </span><p>Hello, {{ name }}!</p> </span></body> </span></html> </span>"#</span>, </span> ) </span> .</span>unwrap</span>(); </span> </span> </span>let</span> with_tera </span>= </span>warp::any().</span>map</span>(</span>move \|\|</span> tera.</span>clone</span>()); </span> </span> </span>let</span> hello </span>= </span>path!(</span>"hello" </span>/ </span>String</span>) </span> .</span>and</span>(with_tera) </span> .</span>map</span>(\|</span>name</span>: String, </span>tera</span>: Tera\| { </span> </span>let </span>mut</span> context </span>= </span>Context::new(); </span> context.</span>insert</span>(</span>"name"</span>, </span>&</span>name); </span> </span> </span>let</span> body </span>=</span> tera.</span>render</span>(</span>"hello.html"</span>, </span>&</span>context).</span>unwrap</span>(); </span> warp::reply::html(body) </span> }); </span> </span> warp::serve(hello).</span>run</span>(([</span>127</span>, </span>0</span>, </span>0</span>, </span>1</span>], </span>3000</span>)).await; </span>} </span></code></pre> Now if we test our hello endpoint again, we should see it returning html instead.</p> curl -i http://localhost:3000/hello/jon </span>HTTP/1.1 200 OK </span>content-type: text/html; charset=utf-8 </span>content-length: 126 </span>date: Sat, 04 Jan 2020 19:59:03 GMT </span> </span> </span><!DOCTYPE html> </span><head> </span> <meta charset="UTF-8"> </span> <title>Hello</title> </span></head> </span><body> </span><p>Hello, jon!</p> </span></body> </span></html> </span></code></pre> Or you could point your browser to http://localhost:3000/hello/name, to see the html rendered.</p> Note</strong></p> You will notice that I am using .unwrap()</code> is several places where the function is returning a Result</a>. I am doing that so that I don't have do to error handling yet. I may write another post about error handling but for now, if you want more information about error handling in rust, this section of the book</a> should help.</p> Login and consent</h3> We showed the basic building blocks for creating a web service in the previous section. Let's put that to use and build our integration to our OAuth 2.0 server. We will implement the login and consent pages and their corresponding form post endpoints.</p> We will first need to update dependencies to add Serde</a> which is the library in Rust to serialize/deserialize to various data formats.</p> Cargo.toml</div> ...</span> </span>[dependencies] </span>...</span> </span>serde </span>= { </span>version </span>= </span>"1.0"</span>, </span>features </span>= [</span>"derive"</span>] } </span></code></pre> With this let's implement the login page and form handler first.</p> src/main.rs</div> pub </span>mod </span>view; </span></code></pre> Will create a view module where we can put our template rendering functions.</p> src/view.rs</div> use </span>tera::Tera; </span>use </span>warp::{self, Filter}; </span> </span>pub </span>fn </span>with_tera</span>() -> warp::filters::BoxedFilter<(Tera,)> { </span> warp::any().</span>map</span>(</span>move \|\| </span>tera_templates</span>()).</span>boxed</span>() </span>} </span> </span>pub </span>fn </span>tera_templates</span>() -> Tera { </span> </span>let</span> login_tpl </span>= </span>r</span>#" </span><!DOCTYPE html> </span><head> </span> <meta charset="UTF-8"> </span> <title>Login</title> </span></head> </span><body> </span><h1>Login page</h1> </span><form action="/login" method="post"> </span>{% if login_challenge %} </span> <input type="hidden" name="login_challenge" value="{{login_challenge}}"/> </span>{% else %} </span>{% endif %} </span> </span> <label for="username">Username</label>: </span> <input type="text" id="username" name="username" autofocus="autofocus"/> <br/> </span> </span> <label for="password">Password</label>: </span> <input type="password" id="password" name="password"/> <br/> </span> </span> <input type="submit" value="Log in"/> </span></form> </span></body> </span></html> </span>"#</span>; </span> </span> </span>let </span>mut</span> tera </span>= </span>Tera::default(); </span> tera.</span>add_raw_templates</span>(vec![(</span>"login.html"</span>, login_tpl)]) </span> .</span>unwrap</span>(); </span> tera </span>} </span></code></pre> And now for the actual login page using the rendering template we just created in the view</code> module.</p> src/main.rs</div> use </span>serde::{Deserialize, Serialize}; </span>use </span>tera::{Context, Tera}; </span>use </span>warp::{self, Filter}; </span> </span>pub </span>mod </span>view; </span> </span>#[tokio::main] </span>async </span>fn </span>main</span>() { </span> </span>let</span> routes </span>= </span>auth_routes</span>(); </span> warp::serve(routes).</span>run</span>(([</span>127</span>, </span>0</span>, </span>0</span>, </span>1</span>], </span>3000</span>)).await; </span>} </span>pub </span>fn </span>auth_routes</span>() -> warp::filters::BoxedFilter<(impl warp::reply::Reply,)> { </span> warp::path(</span>"login"</span>) </span> .</span>and</span>(</span>login_page</span>()) </span> .</span>boxed</span>() </span>} </span> </span> </span>pub </span>fn </span>login_page</span>() -> warp::filters::BoxedFilter<(impl warp::Reply,)> { </span> #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] </span> </span>struct </span>LoginQueryParams { </span> login_challenge: </span>Option</span><</span>String</span>>, </span> } </span> </span> warp::get() </span> .</span>and</span>(warp::query::query()) </span> .</span>and</span>(view::with_tera()) </span> .</span>map</span>(</span>move \|</span>query_params: LoginQueryParams, tera: Tera</span>\| </span>{ </span> </span>let </span>mut</span> context </span>= </span>Context::new(); </span> context.</span>insert</span>(</span>"login_challenge"</span>, </span>&</span>query_params.login_challenge); </span> </span> </span>let</span> body </span>=</span> tera.</span>render</span>(</span>"login.html"</span>, </span>&</span>context).</span>unwrap</span>(); </span> warp::reply::html(body) </span> }) </span> .</span>boxed</span>() </span>} </span></code></pre> If we submit the login form from the login page, we should receive a 404. To fix that we need to implement the form post handling endpoint.</p> src/main.rs</div> use </span>serde::{Deserialize, Serialize}; </span>use </span>tera::{Context, Tera}; </span>use </span>warp::{self, Filter}; </span> </span>pub </span>mod </span>view; </span> </span>... </span> </span>pub </span>fn </span>auth_routes</span>() -> warp::filters::BoxedFilter<(impl warp::reply::Reply,)> { </span> warp::path(</span>"login"</span>) </span> .</span>and</span>(</span>login_page</span>().</span>or</span>(</span>accept_login</span>())) </span> .</span>boxed</span>() </span>} </span> </span>... </span> </span>pub </span>fn </span>accept_login</span>() -> warp::filters::BoxedFilter<(impl warp::Reply,)> { </span> #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] </span> </span>struct </span>FormBody { </span> login_challenge: </span>Option</span><</span>String</span>>, </span> username: String, </span> password: String, </span> } </span> </span> warp::post() </span> .</span>and</span>( </span> warp::body::content_length_limit(</span>1024 </span>* </span>32</span>) </span> .</span>and</span>(warp::body::form()) </span> .</span>map</span>(\|</span>form_body</span>: FormBody\| warp::reply::json(</span>&</span>form_body)), </span> ) </span> .</span>boxed</span>() </span>} </span></code></pre> We now have a working login page and login handler which returns the form body back as json. So if you try to login with the username, "username"</code>, and password, "password"</code>, we should receive the following.</p> http://localhost:3000/login</div> { </span> </span>"username"</span>: </span>"username"</span>, </span> </span>"password"</span>: </span>"password"</span>, </span> </span>"login_challenge"</span>: </span>null </span>} </span></code></pre> Following the same pattern, you can add the consent page and post endpoint.</p> You can see what you should end up with here</a></p> Setting up the Hydra services</h3> To be able to test our integration to hydra we will need to run it locally and to do that, the easiest way I found is to run hydra with their docker-compose</a> file. We will then make a few modifications to it. We could run hydra with an in-memory database, but with a local postgres database we will be able to see how the api calls we will be doing to the hydra instance affect its data store.</p> docker-compose.yml</div> version</span>: </span>"3.3" </span>services</span>: </span> </span>postgres</span>: </span> </span>image</span>: </span>postgres:latest </span> </span>ports</span>: </span> - </span>"5432:5432" </span> </span>environment</span>: </span> </span>POSTGRES_PASSWORD</span>: </span>password </span> </span>volumes</span>: </span> - </span>${PWD}/initdb.sh:/docker-entrypoint-initdb.d/init-user-db.sh </span> </span> </span>hydra</span>: </span> </span>depends_on</span>: </span> - </span>hydra-migrate </span> </span>image</span>: </span>oryd/hydra:latest </span> </span>ports</span>: </span> - </span>"4444:4444" </span># Public port </span> - </span>"4445:4445" </span># Admin port </span> - </span>"5555:5555" </span># Port for hydra token user </span> </span>command</span>: </span>serve all --dangerous-force-http </span> </span>environment</span>: </span> </span># https://www.ory.sh/docs/hydra/configuration </span> </span># - LOG_LEVEL=debug </span> - </span>URLS_SELF_ISSUER=http://localhost:4444 </span> - </span>URLS_CONSENT=http://localhost:3000/consent </span> - </span>URLS_LOGIN=http://localhost:3000/login </span> - </span>DSN=postgres://hydra:hello@postgres:5432/hydra?sslmode=disable </span> - </span>SECRETS_SYSTEM=youReallyNeedToChangeThis </span> - </span>SECRETS_COOKIE=youReallyNeedToChangeThisToo </span> - </span>OAUTH2_EXPOSE_INTERNAL_ERRORS=1 </span> - </span>OIDC_SUBJECT_TYPES_SUPPORTED=public,pairwise </span> - </span>OIDC_SUBJECT_TYPE_PAIRWISE_SALT=youReallyNeedToChangeThisToo2 </span> </span>restart</span>: </span>unless-stopped </span> </span> </span>hydra-migrate</span>: </span> </span>depends_on</span>: </span> - </span>postgres </span> </span>image</span>: </span>oryd/hydra:latest </span> </span>environment</span>: </span> - </span>DSN=postgres://hydra:hello@postgres:5432/hydra?sslmode=disable&max_conns=20&max_idle_conns=4 </span> </span>command</span>: </span>migrate sql -e --yes </span> </span>restart</span>: </span>on-failure </span></code></pre> initdb.sh</div> #!/bin/bash </span>set </span>-e </span> </span>psql</span> -v</span> ON_ERROR_STOP=1</span> --username </span>"$POSTGRES_USER"</span> --dbname </span>"$POSTGRES_DB" </span><<-EOSQL </span> CREATE USER hydra WITH PASSWORD 'hello'; </span> CREATE USER auth WITH PASSWORD 'authpassword'; </span> CREATE DATABASE hydra; </span> CREATE DATABASE auth; </span> GRANT ALL PRIVILEGES ON DATABASE hydra TO hydra; </span> GRANT ALL PRIVILEGES ON DATABASE auth TO auth; </span>EOSQL </span></code></pre> The main difference between this docker-compose.yml</code> file and the one provided by in the Hydra repository, is the initdb.sh</code> script that creates 2 databases and the corresponding user and assigns that user to it's database only.</p> Generating the Hydra OpenApi Client</h3> The Hydra server provides public and admin REST APIs</a> to interface with it and allows us to implement the integration. They also provide sdks, for Go and Javascript, but since we are using Rust we will need to implement our own. Luckily they also provide OpenAPI specs</a> and the OpenAPI generator implements a Rust client generator. The code quality isn't great but that should be enough to get us started with our integration and we can always improve it later by implementing our own client using the generated models.</p> Here is the command to generate our rust OpenAPI client.</p> openapi-generator-cli generate -i https://raw.githubusercontent.com/ory/hydra/v1.1.1/docs/api.swagger.json --package-name hydra --library reqwest -g rust -o /local/hydra </span></code></pre> Or even better, instead of installing the openapi-generator-cli</code>, we can use the docker image to do the same thing and let's put it inside a bash script to be able to reuse it if necessary and remember next time we need to regenerate.</p> generate-hydra-api.sh</div> #!/bin/bash </span> </span>set </span>-o</span> errexit </span>set </span>-o</span> pipefail </span>set </span>-o</span> nounset </span>set </span>-x </span> </span>docker run</span> --rm -v </span>${PWD}:/local openapitools/openapi-generator-cli:latest generate</span> -i</span> https://raw.githubusercontent.com/ory/hydra/v1.1.1/docs/api.swagger.json</span> --package-name</span> hydra</span> --library</span> reqwest</span> -g</span> rust</span> -o</span> /local/hydra </span></code></pre> Now we have 2 crates of code, one that we wrote and the other that we generated. We will need to make some space where this new crate can live and compile the together. Cargo provides the workspace functionality for handling multiple crates in the same repository. To do that, we will move the code we have written so far into a sub directory. Let's call that the auth</code> directory.</p> mkdir auth </span>mv Cargo.toml auth </span>mv src auth </span></code></pre> Change the package name in the Cargo.toml</code> to auth to match the directory name.</p> auth/Cargo.toml</div> [package] </span>name </span>= </span>"auth" </span># <--- name to change </span>version </span>= </span>"0.1.0" </span>authors </span>= [</span>"Jonathan Fok Kan <jfokkan@gmail.com>"</span>] </span>edition </span>= </span>"2018" </span> </span>[dependencies] </span>...</span> </span></code></pre> Then create a Cargo.toml</code> file with the 2 sub crates we now have.</p> Cargo.toml</div> [workspace] </span> </span>members </span>= [ </span> </span>"auth"</span>, </span> </span>"hydra"</span>, </span>] </span></code></pre> Lets compile everything and see if the new generated crate works.</p> cargo clean </span>cargo build </span></code></pre> Oh Ho, something's not working. We are getting a compile error.</p> error[E0308]: mismatched types </span> --> hydra/src/apis/admin_api.rs:542:16 </span> \| </span>542 \| if let Some(ref s) = client { </span> \| ^^^^^^^^^^^ ------ this match expression has type `reqwest::Client` </span> \| \| </span> \| expected struct `reqwest::Client`, found enum `std::option::Option` </span> \| </span> = note: expected type `reqwest::Client` </span> found type `std::option::Option<_>` </span> </span>error: aborting due to previous error </span> </span>For more information about this error, try `rustc --explain E0308`. </span>error: could not compile `hydra`. </span>warning: build failed, waiting for other jobs to finish... </span>error: build failed </span></code></pre> Let's check out the file and see what's wrong. The rust compiler is helpful enough to give us the exact location of the error. Well the error looks pretty simple, the client</code> created within the function body at let client = &configuration.client;</code> is shadowing the client</code> from the function parameters. The solution is just as simple, let's just rename the function parameter to client_id</code> since that's actually what is being sent here. The changed lines will be as follows.</p> hydra/src/apis/admin_api.rs</div> ... </span> </span>fn </span>revoke_consent_sessions</span>(</span>&</span>self</span>, </span>subject</span>: </span>&</span>str</span>, </span>client_id</span>: </span>Option</span><</span>&</span>str</span>>) -> </span>Result</span><(), Error> { </span> </span>let</span> configuration: </span>&</span>configuration::Configuration </span>= </span>self.configuration.</span>borrow</span>(); </span> </span>let</span> client </span>= &</span>configuration.client; </span> </span> </span>... </span> </span> req_builder </span>=</span> req_builder.</span>query</span>(</span>&</span>[(</span>"subject"</span>, </span>&</span>subject.</span>to_string</span>())]); </span> </span>if </span>let Some</span>(</span>ref</span> s) </span>=</span> client_id { </span> req_builder </span>=</span> req_builder.</span>query</span>(</span>&</span>[(</span>"client"</span>, </span>&</span>s.</span>to_string</span>())]); </span> } </span> </span> </span>... </span></code></pre> Updating our login and consent implementation</h3> Now that we have our generated hydra client library, we want to integrate this into our auth</code> project. Let's take a look at the code. For reading and getting a quick overview of the code. My preferred way of doing this is by reading the documentation. Even when there is no documentation, Rust generates pretty nice documentation from the public types and provides a fairly intuitive way of navigating and searching the interface. To generate the documentation, Cargo provides the doc</code> command.</p> cargo doc --open </span></code></pre> If your browser didn't open with documentation page, you should be able to find it at ./target/doc/hydra/index.html</code> in the root directory of the project.</p> Once the documentation is opened, you should find the hydra crate which will be in the crates list. Inside, you will find 2 modules, the apis and models modules. Inside the apis module, we will find the structs and traits that will help us communicate with hydra. The AdminApiClient</code> will be the one we care about to implement the login and consent features.</p> impl AdminApiClient </span>pub fn new(configuration: Rc<Configuration>) -> AdminApiClient </span></code></pre> We can see that AdminApiClient</code> has a new function to create a new instance of the struct and it takes a Rc<Configuration></code>. Rc<Configuration></code> is a reference counted pointer to a Configuration struct and the Configuration struct has some fields that helps the ApiClient figure out how to make calls to the hydra instance.</p> Similar to the Tera template renderer, we will create a Warp Filter that will contain the AdminApiClient and pass it to the function handling the endpoints. But first we need to add the dependency to the hydra crate to our main auth crate.</p> auth/Cargo.toml</div> [dependencies] </span>...</span> </span>hydra </span>= { </span>path </span>= </span>"../hydra" </span>} </span>...</span> </span></code></pre> auth/src/main.rs</div> ... </span>use </span>hydra::apis::{configuration::Configuration, AdminApiClient}; </span>use </span>std::rc::Rc; </span> </span>... </span> </span>pub </span>fn </span>with_hydra_api</span>() -> warp::filters::BoxedFilter<(AdminApiClient,)> { </span> warp::any() </span> .</span>map</span>(</span>move \|\| </span>{ </span> </span>let </span>mut</span> configuration </span>= </span>Configuration::new(); </span> configuration.base_path </span>= </span>HYDRA_ADMIN_ADDRESS</span>.</span>to_owned</span>(); </span> AdminApiClient::new(Rc::new(configuration)) </span> }) </span> .</span>boxed</span>() </span>} </span> </span></code></pre> But something's not right, we are getting a compilation error, with rust stating that our Rc<Configuration></code> isn't Send</code>.</p> error[E0277]: `std::rc::Rc<hydra::apis::configuration::Configuration>` cannot be sent between threads safely </span> --> auth/src/main.rs:44:10 </span> \| </span>44 \| .boxed() </span> \| ^^^^^ `std::rc::Rc<hydra::apis::configuration::Configuration>` cannot be sent between threads safely </span> \| </span> = help: within `(hydra::apis::admin_api::AdminApiClient,)`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<hydra::apis::configuration::Configuration>` </span> = note: required because it appears within the type `hydra::apis::admin_api::AdminApiClient` </span> = note: required because it appears within the type `(hydra::apis::admin_api::AdminApiClient,)` </span></code></pre> What is Send? Send</a> is a marker trait that states whether something can be sent between threads (i.e. thread-safe). What can we do? Well not much, it seems that Warp requires state passed into it's filters to be thread safe since it aims to concurrently handle requests with multiple threads but the openapi code generator produced code that can't. We are going to have to modify the generated code just enough to make it thread safe. One way to do that is simply to switch the Rc with Arc</a> which is the thread safe version of Rc.</p> These 2 commands ought to do it.</p> find ./hydra -type f -exec sed -i '' 's/std::rc::Rc/std::sync::Arc/g' {} \; </span>find ./hydra -type f -exec sed -i '' 's/Rc/Arc/g' {} \; </span></code></pre> Now instead of AdminApiClient::new(Rc::new(configuration))</code> we can pass it AdminApiClient::new(Arc::new(configuration))</code>.</p> Let's implement the accept login request logic for the hydra integration in the login page.</p> auth/src/main.rs</div> use </span>log::info; </span>use </span>serde::{Deserialize, Serialize}; </span>use </span>tera::{Context, Tera}; </span>use </span>warp::{self, http::Uri, Filter}; </span> </span>use </span>hydra::apis::{configuration::Configuration, AdminApi, AdminApiClient}; </span>use </span>std::{</span>str</span>::FromStr, sync::Arc}; </span>... </span> </span>pub </span>fn </span>login_page</span>() -> warp::filters::BoxedFilter<(impl warp::Reply,)> { </span> #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] </span> </span>struct </span>LoginQueryParams { </span> login_challenge: </span>Option</span><</span>String</span>>, </span> } </span> </span> warp::get() </span> .</span>and</span>(warp::query::query()) </span> .</span>and</span>(view::with_tera()) </span> .</span>and</span>(</span>with_hydra_api</span>()) </span> .</span>map</span>( </span> </span>move \|</span>query_params: LoginQueryParams, tera: Tera, hydra_api: AdminApiClient</span>\| </span>{ </span> </span> </span>// The challenge is used to fetch information about the login request from ORY Hydra. </span> query_params </span> .login_challenge </span> .</span>map</span>(\|</span>login_challenge</span>\| { </span> </span>let</span> login_request </span>= </span> hydra_api.</span>get_login_request</span>(</span>&</span>login_challenge).</span>unwrap</span>(); </span> </span> </span>// If hydra was already able to authenticate the user, skip will be true and we do not need to re-authenticate </span> </span>if</span> login_request.skip.</span>unwrap_or</span>(</span>false</span>) { </span> info!(</span>"Hydra was already able to authenticate the user, skipping login as we do not need to re-authenticate"</span>); </span> info!(</span>"Accepting login request with Hydra"</span>); </span> </span> </span>// You can apply logic here, for example update the number of times the user logged in. </span> </span>// ... </span> </span> </span>// Now it's time to grant the login request. You could also deny the request if something went terribly wrong </span> </span>// (e.g. your arch-enemy logging in...) </span> </span>let</span> completed_request </span>=</span> hydra_api </span> .</span>accept_login_request</span>( </span> </span>&</span>login_challenge, </span> </span>Some</span>(hydra::models::AcceptLoginRequest::new( </span> </span>// All we need to do is to confirm that we indeed want to log in the user. </span> </span>// We are using a hardcoded subject here, the subject should be an immutable id of the user that is loggin in </span> </span>// to let Hydra know which user to associate with this login. </span> </span>"hardcoded_subject"</span>.</span>to_owned</span>(), </span> )), </span> ) </span> .</span>unwrap</span>(); </span> </span> </span>// All we need to do now is to redirect the user back to hydra! </span> </span>Box</span>::new(warp::redirect( </span> Uri::from_str( </span> </span>&</span>completed_request </span> .redirect_to </span> .</span>unwrap_or</span>(</span>"/"</span>.</span>to_owned</span>()), </span> ) </span> .</span>unwrap</span>(), </span> )) </span>as </span>Box</span><dyn warp::Reply> </span> } </span>else </span>{ </span> </span>// If authentication can't be skipped we MUST show the login UI. </span> info!(</span>"Sending user to login"</span>); </span> </span> </span>// The challenge will be a hidden input field </span> </span>let </span>mut</span> context </span>= </span>Context::new(); </span> context.</span>insert</span>(</span>"login_challenge"</span>, </span>&</span>login_challenge); </span> </span> </span>let</span> body </span>=</span> tera.</span>render</span>(</span>"login.html"</span>, </span>&</span>context).</span>unwrap</span>(); </span> </span>Box</span>::new(warp::reply::html(body)) </span>as </span>Box</span><dyn warp::Reply> </span> } </span> }) </span> .</span>unwrap_or_else</span>(\|\| { </span> </span>let</span> body </span>=</span> tera.</span>render</span>(</span>"login.html"</span>, </span>&</span>Context::new()).</span>unwrap</span>(); </span> </span>Box</span>::new(warp::reply::html(body)) </span>as </span>Box</span><dyn warp::Reply> </span> }) </span> }, </span> ) </span> .</span>boxed</span>() </span>} </span></code></pre> There is quite a bit happening here, so let's explain. First, I added info logging in a few places, which is why I added the log::info</code> dependency. We will need to turn on the logs later on, but let's not worry about that for now. The logic goes as follows:</p> We map the login_challenge</code> from the query parameters, if it doesn't exist, we render a login page without login challenge. I did this because I expected the login page to also be usable outside of an oauth2 flow, in which case we could simply return a session cookie once the user is authenticated. We could also have simply returned an error if the login challenge wasn't there.</li> If the login challenge exists, we call hydra's get login request api and with the login request we check if we could skip the login page if the user had successfully authenticated with this browser in the past.</li> If we can skip the user authentication, we go straight into accepting the login request with the hydra api. This is also where we could have added some logic before calling the accept login api to do various checks.</li> Once the login request is accepted, we redirect to the url provided by hydra or to the homepage if it's not provided. In this case since we are expecting to be redirected by hydra, we could also have returned an error page if the redirect url was empty.</li> If we can't skip authentication, we send the user to the login page with the login challenge set.</li> </ol> Something else you may have noticed is that instead of returning a warp::Reply</code> directly, I wrapped it inside of a Box so that our function is returning a trait object. This is because we are returning 2 different types both implementing the Reply trait in the 2 branches of our if else expression. This wouldn't work unless we make them both the same type by turning them into trait objects. To learn more about Trait Objects check out these links</a>.</p> I also imported the std::str::FromStr</code> trait so that it can be used to convert an &str</code> to a Uri</code>.</p> As for handling the login form, we will be following similar logic as above.</p> auth/src/main.rs</div> ... </span> </span>pub </span>fn </span>accept_login</span>() -> warp::filters::BoxedFilter<(impl warp::Reply,)> { </span> #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] </span> </span>struct </span>FormBody { </span> login_challenge: </span>Option</span><</span>String</span>>, </span> username: String, </span> password: String, </span> } </span> </span> warp::post() </span> .</span>and</span>( </span> warp::body::content_length_limit(</span>1024 </span>* </span>32</span>) </span> .</span>and</span>(warp::body::form()) </span> .</span>and</span>(</span>with_hydra_api</span>()) </span> .</span>map</span>(\|</span>form_body</span>: FormBody, </span>hydra_api</span>: AdminApiClient\| { </span> </span>// Add logic here to verify the username and password from the submitted login form </span> </span> </span>// Accepting login request, although you could still deny the login request if something else went wrong </span> form_body </span> .login_challenge </span> .</span>map</span>(\|</span>login_challenge</span>\| { </span> info!(</span>"Accepting login request with Hydra"</span>); </span> </span>let</span> completed_request </span>=</span> hydra_api </span> .</span>accept_login_request</span>( </span> </span>&</span>login_challenge, </span> </span>Some</span>(hydra::models::AcceptLoginRequest::new( </span> </span>// We are using a hardcoded subject here, the subject should be an immutable id of the user that is loggin in </span> </span>// to let Hydra know which user to associate with this login </span> </span>"hardcoded_subject"</span>.</span>to_owned</span>(), </span> )), </span> ) </span> .</span>unwrap</span>(); </span> </span> </span>// Redirecting to hydra </span> warp::redirect( </span> Uri::from_str( </span> </span>&</span>completed_request.redirect_to.</span>unwrap_or</span>(</span>"/"</span>.</span>to_owned</span>()), </span> ) </span> .</span>unwrap</span>(), </span> ) </span> }) </span> .</span>unwrap_or_else</span>(\|\| warp::redirect(Uri::from_str(</span>"/"</span>).</span>unwrap</span>())) </span> }), </span> ) </span> .</span>boxed</span>() </span>} </span></code></pre> And following the same principles as the login logic, we can do the same for the consent.</p> Enabling logging</h4> Before we finish, let's print the logs that we inserted. To do that we will be using the env_logger library. It's a fairly simple library that allows you to control the log level and which crates should have it's logs enabled through an environment variable, by default that env var is RUST_LOG</code>.</p> I will also add the log Filter from warp to give us access logs on our endpoints.</p> auth/src/main.rs</div> #[tokio::main] </span>async </span>fn </span>main</span>() { </span> </span>if </span>::std::env::var_os(</span>"RUST_LOG"</span>).</span>is_none</span>() { </span> ::std::env::set_var(</span>"RUST_LOG"</span>, </span>"warp=info,auth_svc=trace,api_access=trace"</span>); </span> } </span> env_logger::init(); </span> </span> </span>let</span> routes </span>= </span>auth_routes</span>().</span>or</span>(</span>homepage</span>()).</span>with</span>(warp::log(</span>"api_access"</span>)); </span> warp::serve(routes).</span>run</span>(([</span>127</span>, </span>0</span>, </span>0</span>, </span>1</span>], </span>3000</span>)).await; </span>} </span></code></pre> What you should now have.</a></p> Testing our login and consent integration</h3> We now have working login and consent application and a way to deploy hydra locally. How do we know what we wrote is work? Let's try it out manually and then write some tests for it.</p> docker-compose up -d </span>cargo run </span></code></pre> Create a client in Hydra.</p> curl -X POST \ </span> http://localhost:4445/clients \ </span> -H 'Content-Type: application/json' \ </span> -d '{ </span> "client_id": "my-implicit-client", </span> "grant_types": ["implicit"], </span> "response_types": ["token"], </span> "redirect_uris": ["http://localhost:3000/"], </span> "token_endpoint_auth_method": "client_secret_post" </span>}' </span></code></pre> To start the flow, navigate to Authorization Initialization link</a> which is usually generated by an OAuth2 client or library.</p> After going through the login and consent pages, we should end up back on the welcome page with the access token.</p> http://localhost:3000/#access_token=IQc9NKSJyHS9Vy9iS05kyXVoUTkXexCQPxq-6_Ly5C8.m5m7P-RYhty0o47x35D1uF-k_JJdttzjxqqw11Kr22M&expires_in=3600&scope=offline&state=blahblahblah&token_type=bearer </span></code></pre> Since we know that our integration is working, let's write a test that we can run to verify that it keeps working in the future. To do that, we will create a tests</code> directory in our auth</code> project. That's where integration tests usually go, whereas unit tests can be written inline (For more information on tests check out these links</a>). But we might want to refer to some structs from our crate such as the form model structs, currently we can't do that because we wrote all our code in a main.rs</code> file which is used for creating a binary. We would need to move the code that we would like to be able to share as a library into a lib.rs</code> file.</p> We will add a function that returns our routes which can then be used from the main.rs</code> file.</p> auth/src/lib.rs</div> ... </span>pub </span>fn </span>routes</span>() -> warp::filters::BoxedFilter<(impl warp::reply::Reply,)> { </span> </span>auth_routes</span>() </span> .</span>or</span>(</span>homepage</span>()) </span> .</span>with</span>(warp::log(</span>"api_access"</span>)) </span> .</span>boxed</span>() </span>} </span>... </span></code></pre> auth/src/main.rs</div> use</span> auth; </span>use</span> warp; </span> </span>#[tokio::main] </span>async </span>fn </span>main</span>() { </span> </span>if </span>::std::env::var_os(</span>"RUST_LOG"</span>).</span>is_none</span>() { </span> ::std::env::set_var(</span>"RUST_LOG"</span>, </span>"warp=info,auth_svc=trace,api_access=trace"</span>); </span> } </span> env_logger::init(); </span> </span> warp::serve(auth::routes()) </span> .</span>run</span>(([</span>127</span>, </span>0</span>, </span>0</span>, </span>1</span>], </span>3000</span>)) </span> .await; </span>} </span></code></pre> Then we will have to expose the structs we created to model the form bodies of our login and consent apis.</p> auth/src/lib.rs</div> ... </span>#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] </span>pub </span>struct </span>LoginFormBody { </span> </span>pub </span>login_challenge: </span>Option</span><</span>String</span>>, </span> </span>pub </span>username: String, </span> </span>pub </span>password: String, </span>} </span> </span>... </span> </span>#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] </span>pub </span>enum </span>ConsentStatus { </span> Authorize, </span> Deny, </span>} </span> </span>#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] </span>pub </span>struct </span>ConsentFormBody { </span> </span>pub </span>consent_challenge: String, </span> </span>pub </span>submit: ConsentStatus, </span>} </span>... </span></code></pre> Now that we have a library crate we can get to the actual work of writing the test.</p> auth/Cargo.toml</div> ...</span> </span>[dev-dependencies] </span>uuid </span>= { </span>version </span>= </span>"0.8"</span>, </span>features </span>= [</span>"v4"</span>] } </span>oauth2 </span>= </span>" 3.0.0-alpha.7" </span>reqwest </span>= { </span>version </span>= </span>"0.10.0"</span>, </span>features </span>= [</span>"json"</span>, </span>"blocking"</span>, </span>"cookies"</span>] } </span></code></pre> The body of a basic test is as follows.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> #[test] </span>fn </span>basic_test</span>() { </span> assert!(</span>true</span>); </span>} </span></code></pre> Since this is an integration test we may want to check that our external dependencies are up.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> use </span>reqwest::{self, Url}; </span> </span>#[test] </span>fn </span>login_and_consent_flow</span>() { </span> </span>check_auth_svc</span>(</span>AUTHN_BASE_URL</span>); </span> </span>check_hydra</span>(</span>HYDRA_ADMIN_BASE_URL</span>); </span>} </span> </span>fn </span>check_auth_svc</span>(</span>authn_url</span>: </span>&</span>str</span>) { </span> </span>let</span> res </span>= </span>reqwest::blocking::get(authn_url).</span>unwrap</span>(); </span> assert_eq!(res.</span>status</span>(), reqwest::StatusCode::</span>OK</span>); </span>} </span> </span>fn </span>check_hydra</span>(</span>hydra_url</span>: </span>&</span>str</span>) { </span> </span>let</span> hydra_url </span>= </span>Url::parse(hydra_url).</span>unwrap</span>(); </span> </span>let</span> res </span>= </span>reqwest::blocking::get(hydra_url.</span>join</span>(</span>"/health/ready"</span>).</span>unwrap</span>()).</span>unwrap</span>(); </span> assert_eq!(res.</span>status</span>(), reqwest::StatusCode::</span>OK</span>); </span>} </span></code></pre> As we saw in our manual test, we will first need to create an oauth2 client to do our test. To know which type of oauth2 client we want to create, we should decide what we will be testing. So let's say we will be testing the authorization code grant type.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> use </span>hydra::apis::{configuration::Configuration, AdminApi, AdminApiClient}; </span>use </span>reqwest::{self, Url}; </span>use </span>std::sync::Arc; </span>use </span>uuid::Uuid; </span> </span>#[test] </span>fn </span>login_and_consent_flow</span>() { </span> </span>... </span> </span> </span>let </span>mut</span> config </span>= </span>Configuration::new(); </span> config.base_path </span>= </span>HYDRA_ADMIN_BASE_URL</span>.</span>to_owned</span>(); </span> </span>let</span> hydra_admin_client </span>= </span>AdminApiClient::new(Arc::new(config)); </span> </span> </span>let</span> oauth2_client </span>= </span>create_oauth2_client</span>(</span>&</span>hydra_admin_client); </span>} </span> </span>fn </span>create_oauth2_client</span>(</span>hydra_admin_client</span>: </span>&</span>AdminApiClient) -> hydra::models::OAuth2Client { </span> </span>let </span>mut</span> new_oauth2_client </span>= </span>hydra::models::OAuth2Client::new(); </span> new_oauth2_client.client_id </span>= </span>Some</span>(format!( </span> </span>"</span>{}</span>-</span>{}</span>"</span>, </span> Uuid::new_v4().</span>to_string</span>(), </span> </span>"my-test-client" </span> )); </span> new_oauth2_client.client_name </span>= </span>Some</span>(</span>"login-flow-test-client"</span>.</span>to_owned</span>()); </span> new_oauth2_client.client_secret </span>= </span>Some</span>(</span>"client-secret"</span>.</span>to_owned</span>()); </span> new_oauth2_client.grant_types </span>= </span>Some</span>(vec![</span>"authorization_code"</span>.</span>to_owned</span>()]); </span> new_oauth2_client.redirect_uris </span>= </span>Some</span>(vec![</span>AUTHN_BASE_URL</span>.</span>to_owned</span>()]); </span> new_oauth2_client.token_endpoint_auth_method </span>= </span>Some</span>(</span>"client_secret_basic"</span>.</span>to_owned</span>()); </span> new_oauth2_client.scope </span>= </span>Some</span>(</span>"openid"</span>.</span>to_owned</span>()); </span> </span> hydra_admin_client </span> .</span>create_o_auth2_client</span>(new_oauth2_client) </span> .</span>unwrap</span>() </span>} </span></code></pre> With the created OAuth 2 client, we can initiate an oauth2 flow. For that, we will use an OAuth2 library</a>.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> </span>#[test] </span>fn </span>login_and_consent_flow</span>() { </span> </span>... </span> </span> </span>let</span> access_token </span>= </span>initiate_oauth2_code_flow</span>( </span> </span>&</span>oauth2_client.client_id.</span>unwrap</span>(), </span> </span>&</span>oauth2_client.client_secret.</span>unwrap</span>(), </span> ); </span>} </span> </span>... </span> </span>fn </span>initiate_oauth2_code_flow</span>(</span>client_id</span>: </span>&</span>str</span>, </span>client_secret</span>: </span>&</span>str</span>) -> String { </span> </span>// Create an HTTP client to act as the browser </span> </span>let</span> reqwest_client </span>= </span>reqwest::blocking::Client::builder() </span> </span>// Enable the cookie store for hydra csrf cookies </span> .</span>cookie_store</span>(</span>true</span>) </span> .</span>redirect</span>(reqwest::redirect::Policy::none()) </span> .</span>build</span>() </span> .</span>unwrap</span>(); </span> </span> </span>// Create an OAuth2 client by specifying the client ID, client secret, authorization URL and </span> </span>// token URL. </span> </span>let</span> client </span>= </span>OAuth2BasicClient::new( </span> ClientId::new(client_id.</span>to_owned</span>()), </span> </span>Some</span>(ClientSecret::new(client_secret.</span>to_owned</span>())), </span> AuthUrl::new(</span>"http://localhost:4444/oauth2/auth"</span>.</span>to_owned</span>()).</span>unwrap</span>(), </span> </span>Some</span>(TokenUrl::new(</span>"http://localhost:4444/oauth2/token"</span>.</span>to_owned</span>()).</span>unwrap</span>()), </span> ) </span> </span>// Set the URL the user will be redirected to after the authorization process. </span> .</span>set_redirect_url</span>(RedirectUrl::new(</span>"http://localhost:3000/"</span>.</span>to_string</span>()).</span>unwrap</span>()); </span> </span> </span>// Generate a PKCE challenge. </span> </span>let </span>(pkce_challenge, pkce_verifier) </span>= </span>PkceCodeChallenge::new_random_sha256(); </span> </span> </span>// Generate the full authorization URL. </span> </span>let </span>(auth_url, csrf_token) </span>=</span> client </span> .</span>authorize_url</span>(CsrfToken::new_random) </span> </span>// Set the desired scopes. </span> .</span>add_scope</span>(Scope::new(</span>"openid"</span>.</span>to_string</span>())) </span> </span>// Set the PKCE code challenge. </span> .</span>set_pkce_challenge</span>(pkce_challenge) </span> .</span>url</span>(); </span> </span> </span>// This is the URL you should usually redirect the user to, </span> </span>// in order to trigger the authorization process. </span> println!(</span>"URL to trigger authorization process: </span>\n{}</span>"</span>, auth_url); </span>} </span></code></pre> Usually the authorize url would then be used to redirect the user (resource owner) to initiate the authorization process. But since we are writing an automated test, we will be emulating the browser with an http client and call our login and consent apis directly.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> </span>fn </span>initiate_oauth2_code_flow</span>(</span>client_id</span>: </span>&</span>str</span>, </span>client_secret</span>: </span>&</span>str</span>) -> String { </span> </span>... </span> </span> </span>let</span> hydra_login_completed_url </span>= </span>perform_login_flow</span>(</span>&</span>reqwest_client, </span>&</span>auth_url); </span>} </span> </span>fn </span>perform_login_flow</span>(</span>reqwest_client</span>: </span>&</span>reqwest::blocking::Client, </span>auth_url</span>: </span>&</span>Url) -> String { </span> </span>let</span> hydra_authz_res </span>=</span> reqwest_client.</span>get</span>(auth_url.</span>as_ref</span>()).</span>send</span>().</span>unwrap</span>(); </span> println!(</span>"</span>\n</span>This is the response from Hydra when visiting the url to trigger the authorization process"</span>); </span> dbg!(</span>&</span>hydra_authz_res); </span> </span> assert!(hydra_authz_res.</span>status</span>().</span>is_redirection</span>()); </span> </span> </span>let</span> login_url </span>=</span> hydra_authz_res </span> .</span>headers</span>() </span> .</span>get</span>(reqwest::header::</span>LOCATION</span>) </span> .</span>unwrap</span>() </span> .</span>to_str</span>() </span> .</span>unwrap</span>(); </span> </span>let</span> login_challenge </span>= </span>Url::parse(login_url) </span> .</span>unwrap</span>() </span> .</span>query_pairs</span>() </span> .</span>find</span>(\|</span>c</span>\| c.</span>0.</span>as_ref</span>().</span>eq_ignore_ascii_case</span>(</span>"login_challenge"</span>)) </span> .</span>unwrap</span>() </span> .</span>1 </span> .</span>to_owned</span>() </span> .</span>to_string</span>(); </span> </span> </span>let</span> login_res </span>=</span> reqwest_client </span> .</span>post</span>(</span>"http://localhost:3000/login"</span>) </span> .</span>form</span>(</span>&</span>LoginFormBody { </span> login_challenge: </span>Some</span>(login_challenge), </span> username: </span>"username"</span>.</span>to_owned</span>(), </span> password: </span>"pass"</span>.</span>to_owned</span>(), </span> }) </span> .</span>send</span>() </span> .</span>unwrap</span>(); </span> </span> println!(</span>"</span>\n</span>This is the response from our auth service when visiting the login page"</span>); </span> dbg!(</span>&</span>login_res); </span> </span> assert!(login_res.</span>status</span>().</span>is_redirection</span>()); </span> </span> </span>let</span> hydra_redirected_url </span>=</span> login_res </span> .</span>headers</span>() </span> .</span>get</span>(reqwest::header::</span>LOCATION</span>) </span> .</span>unwrap</span>() </span> .</span>to_str</span>() </span> .</span>unwrap</span>(); </span> </span> hydra_redirected_url.</span>to_owned</span>() </span>} </span></code></pre> You can note that I disabled redirection in our http client so that we can see all the redirects that are occurring. That is not strictly necessary and we could simply have followed the redirects which would also have made our code much shorter.</p> Same thing for the consent.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> </span>fn </span>initiate_oauth2_code_flow</span>(</span>client_id</span>: </span>&</span>str</span>, </span>client_secret</span>: </span>&</span>str</span>) -> String { </span> </span>... </span> </span> </span>let</span> redirected_url </span>= </span>perform_consent_flow</span>(</span>&</span>reqwest_client, </span>&</span>hydra_login_completed_url); </span>} </span> </span>... </span> </span>fn </span>perform_consent_flow</span>( </span> </span>reqwest_client</span>: </span>&</span>reqwest::blocking::Client, </span> </span>hydra_login_completed_url</span>: </span>&</span>str</span>, </span>) -> String { </span> </span>let</span> hydra_consent_res </span>=</span> reqwest_client </span> .</span>get</span>(hydra_login_completed_url) </span> .</span>send</span>() </span> .</span>unwrap</span>(); </span> </span> println!(</span>"</span>\n</span>This is the response from hydra when visiting the url received at the end of the login process"</span>); </span> dbg!(</span>&</span>hydra_consent_res); </span> println!(</span>"</span>\n</span>As we can see, hydra will now redirect us to our consent page since the login request was accepted"</span>); </span> </span> assert!(hydra_consent_res.</span>status</span>().</span>is_redirection</span>()); </span> </span> </span>let</span> consent_url </span>=</span> hydra_consent_res </span> .</span>headers</span>() </span> .</span>get</span>(reqwest::header::</span>LOCATION</span>) </span> .</span>unwrap</span>() </span> .</span>to_str</span>() </span> .</span>unwrap</span>(); </span> </span>let</span> consent_challenge </span>= </span>Url::parse(consent_url) </span> .</span>unwrap</span>() </span> .</span>query_pairs</span>() </span> .</span>find</span>(\|</span>c</span>\| c.</span>0.</span>as_ref</span>().</span>eq_ignore_ascii_case</span>(</span>"consent_challenge"</span>)) </span> .</span>unwrap</span>() </span> .</span>1 </span> .</span>to_owned</span>() </span> .</span>to_string</span>(); </span> </span> </span>let</span> consent_res </span>=</span> reqwest_client </span> .</span>post</span>(</span>"http://localhost:3000/consent"</span>) </span> .</span>form</span>(</span>&</span>ConsentFormBody { </span> consent_challenge: consent_challenge, </span> submit: ConsentStatus::Authorize, </span> }) </span> .</span>send</span>() </span> .</span>unwrap</span>(); </span> </span> println!(</span>"</span>\n</span>This is the response from our auth service when visiting the consent page"</span>); </span> dbg!(</span>&</span>consent_res); </span> </span> assert!(consent_res.</span>status</span>().</span>is_redirection</span>()); </span> </span> </span>let</span> hydra_consent_approved_url </span>=</span> consent_res </span> .</span>headers</span>() </span> .</span>get</span>(reqwest::header::</span>LOCATION</span>) </span> .</span>unwrap</span>() </span> .</span>to_str</span>() </span> .</span>unwrap</span>(); </span> </span> </span>let</span> hydra_consent_approved_res </span>=</span> reqwest_client </span> .</span>get</span>(hydra_consent_approved_url) </span> .</span>send</span>() </span> .</span>unwrap</span>(); </span> println!(</span>"</span>\n</span>This is the response from hydra once the consent request was accepted. You can note that it contains the authorization code appended to the redirect url configured at the start of the OAuth2 process"</span>); </span> dbg!(</span>&</span>hydra_consent_approved_res); </span> </span> </span>let</span> redirected_url </span>=</span> hydra_consent_approved_res </span> .</span>headers</span>() </span> .</span>get</span>(reqwest::header::</span>LOCATION</span>) </span> .</span>unwrap</span>() </span> .</span>to_str</span>() </span> .</span>unwrap</span>(); </span> </span> redirected_url.</span>to_owned</span>() </span>} </span></code></pre> With the final redirected url, we will receive an auth code that is used in the authorization code grant type to get our access token.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> </span>fn </span>initiate_oauth2_code_flow</span>(</span>client_id</span>: </span>&</span>str</span>, </span>client_secret</span>: </span>&</span>str</span>) -> String { </span> </span>let</span> auth_code </span>= </span>Url::parse(</span>&</span>redirected_url) </span> .</span>unwrap</span>() </span> .</span>query_pairs</span>() </span> .</span>find</span>(\|</span>c</span>\| c.</span>0.</span>as_ref</span>().</span>eq_ignore_ascii_case</span>(</span>"code"</span>)) </span> .</span>unwrap</span>() </span> .</span>1 </span> .</span>to_owned</span>() </span> .</span>to_string</span>(); </span> </span> </span>// Once the user has been redirected to the redirect URL, you'll have access to the </span> </span>// authorization code. For security reasons, your code should verify that the `state` </span> </span>// parameter returned by the server matches `csrf_state`. </span> </span>// Here we are skipping this check for the purposes of brevity of this test. </span> </span> </span>// Now you can trade it for an access token. </span> </span>let</span> token_result </span>=</span> client </span> .</span>exchange_code</span>(AuthorizationCode::new(auth_code.</span>to_string</span>())) </span> </span>// Set the PKCE code verifier. </span> .</span>set_pkce_verifier</span>(pkce_verifier) </span> .</span>request</span>(http_client) </span> .</span>unwrap</span>(); </span> println!(</span>"</span>\n</span>Once the authorization code is exchanged, we finally received an access token"</span>); </span> dbg!(</span>&</span>token_result); </span> println!(</span>"</span>{:?}</span>"</span>, token_result.</span>access_token</span>().</span>secret</span>()); </span> token_result.</span>access_token</span>().</span>secret</span>().</span>to_owned</span>() </span>} </span></code></pre> Now that we have an access token, we should probably see if we can verify it with the introspection endpoint</a>.</p> auth/tests/oauth2_auth_code_end_to_end_test.rs</div> #[test] </span>fn </span>login_and_consent_flow</span>() { </span> </span>... </span> </span> </span>let</span> access_token </span>= </span>initiate_oauth2_code_flow</span>( </span> </span>&</span>oauth2_client.client_id.</span>unwrap</span>(), </span> </span>&</span>oauth2_client.client_secret.</span>unwrap</span>(), </span> ); </span> </span> </span>introspect_access_token</span>(</span>&</span>hydra_admin_client, </span>&</span>access_token); </span>} </span> </span>... </span> </span>fn </span>introspect_access_token</span>(</span>hydra_admin_client</span>: </span>&</span>AdminApiClient, </span>access_token</span>: </span>&</span>str</span>) { </span> </span>let</span> introspection_res </span>=</span> hydra_admin_client </span> .</span>introspect_o_auth2_token</span>(access_token, </span>None</span>) </span> .</span>unwrap</span>(); </span> dbg!(</span>&</span>introspection_res); </span> assert!(introspection_res.active); </span>} </span></code></pre> We can now run our test to see everything running. Since I added some print statements, to be able to see those we will also need to run the test command with the --nocapture</code> flag.</p> cargo test -- --nocapture </span></code></pre> Code with the test</a></p> Async Rust</h3> If you have kept up to date with Async Rust, you may have noticed that the hydra generated client uses the blocking version of the Reqwest library to make it's api calls. This means that by making these API calls we are blocking the tokio executor. What this means is that in cases of load on our auth server, the server may have all it's executor threads blocked on the blocking call we are currently making and prevent the server from serving as many requests as it could.</p> There are 2 ways to fix, the easiest would be to move the blocking call into a thread where blocking isn't an issue. To do that we could spawn the blocking call on a seperate thread where we wouldn't block the tokio executor. See these links for more info</a>.</p> The second would be to rewrite the blocking api client with a non-blocking http client. The latest version of reqwest is such a client.</p> Where to go from here?</h2> We now have a working OAuth 2.0 authorization server integrated with a bare bones auth server. Having a working OAuth server is only the start of implementing a good authorization system for a web service. Integrating it with your services, making access control decisions, how to pass authorization information to your resource servers are only some of the things that still need to be done to get a working system. We also took several shortcuts in this demo to keep the code short and focus on the parts we care about more but there are also several things that need to be done to make our service production ready. Here are several things I can see need improvement in no particular order:</p> Error handling, currently our service will panic at anything deviating from the happy path</li> Add CSRF protection to our login and consent pages and endpoints. This is necessary to prevent a class of attacks called Login CSRF</a></li> Add User management features to our auth service so that it can implement authentication logic</li> Add actual authentication logic to the login endpoints</li> Deploy our service to kubernetes. Hydra already provides Helm Charts for that</a></li> Making sure to follow Hydra's production guide before deploying our services. Link</a></li> Put our services behind an API Gateway or reverse proxy so that all our services are available at one address</li> Add https certificates to our services, this may be easier with a reverse proxy/API Gateway. This is necessary because OAuth 2 simply doesn't have any protection without it</li> Protect a web service's resources(a resource server) with our authorization server</li> Integrate with Ory's other projects such as Keto</a> to provide Access Control rules processing</li> </ul> References</h2> </div> RFC 6749: The OAuth 2.0 Authorization Framework</a></span></li> List of IETF OAuth 2.0 RFCs</a></span></li> Links to various documents about OAuth 2.0</a> from oauth.net</span></li> The latest security best practices for OAuth 2.0</a></span></li> OAuth 2.0 Simplified</a> by Aaron Parecki</span></li> OAuth 2.0 OpenID Connect</span> OpenID Connect Explained</a></li> An OpenID Connect Primer from Okta Identity, Claims, & Tokens: Part 1</a></li> OIDC In Action: Part 2</a></li> What's in a Token?: Part 3</a></li> </ol> </li> </ol> </li> A previous Hydra integration I wrote with Spring and Java Git Repo</a></span></li> Information about when you may want to use different types of access control (Access Tokens vs session cookies)</a></span></li> Ory Hydra REST API docs</span> Ory Hydra REST API docs</a></li> OryOS.14 Versioned Github link</a></li> </ol> </li> OpenAPI</span> OpenAPI Generator</a></li> Github Link</a></li> </ol> </li> Rust Send and Sync Traits</span> Send std docs</a></li> Send and Sync from the Rustonomicon</a></li> </ol> </li> Rust Trait Objects</span> Generic Data Types</a></li> Trait Objects</a></li> </ol> </li> Rust Testing</span> Writing Automated Tests from the Rust Book</a></li> Testing from Rust By Example</a></li> </ol> </li> OAuth 2.0 Token Introspection Extension</span> OAuth2 Token Introspection Endpoint from oauth.com</a></li> RFC 7662: OAuth 2.0 Token Introspection</a></li> Ory Hydra Introspection docs</a></li> </ol> </li> Async Rust: Tokio Blocking</span> tokio::task</a></li> tokio::task::block_in_place</a></li> tokio::task::spawn_blocking</a></li> </ol> </li> </ol> Go and JSON, various useful libraries and utilities Unknown — Fri, 12 Aug 2016 00:00:00 +0000 JSON is probably one of the most popular data interchange format on the web. Go with its extensive standard library has good support for JSON. If your requirements for JSON serialization and deserialization are not specific, I highly recommend to simply start by using "encoding/json"</code> from the standard library and reading how it works (blog</a>, docs</a>, examples</a>). Here I will give a few libraries for JSON in Go and explain what differentiates them from "encoding/json" and each other.</p> First, why would we want to use anything other than "encoding/json"</code>? Well to provide flexibility and simplicity in its usage, "encoding/json"</code> makes use of reflection on the structs that defines the structure of the JSON you are expecting to serialize or deserialize and because of that it incurs a heavy performance penalty.</p> What if we didn't have to reflect on a struct and simply insert data to a map[string]interface{}</code> object? Well "encoding/json"</code> allows you to do that but the resulting data structure ends up frustrating to work with, with casting all over the place and is hard to navigate fluently.</p> Structured Data</h2> ffjson</a></li> easyjson</a></li> </ul> To avoid reflection, these libraries provide a code generator to be run over structs defining the structure of the JSON and fields in the same format as "encoding/json"</code>. The generated code is now the parsing code for the JSON structure.</p> Both ffjson</code> and easyjson</code> provide a drop in replacement for "encoding/json"</code> which means that after generating code for parsing, you should be able to use the same api calls as before but with ffjson</code> or easyjson</code>.</p> The advantage of this is that you now have specific parsing code for the JSON structure you are expecting. This also gives a nice speed up in the order of about 2 to 3 times faster</strong> than encoding/json</code>.</p> The disadvantage is that you will need to run the code generator every time you change the struct defining the JSON structure. There is a simple fix for this by using the go generate</code> command and //go:generate ffjson $GOFILE</code> at the top of any files with the structs.</p> easyjson</code> works very similarly to ffjson</code> and claims to be more performant and behave better when run concurrently.</p> Deserializing Unstructured Data</h2> jsonparser</a></li> gjson</a></li> </ul> jsonparser</code> and gjson</code> like ffjson</code> and easyjson</code> do not rely on reflection but what differentiates them from the structured JSON libraries is that instead of creating structs that defines the structure of the JSON to be parsed these libraries read the value at a specific path you want.</p> By only reading values at paths specified, you only need to parse the part of the data you are interested in. No need to parse the whole json eagerly. Users of lazy functional languages will recognize here the value of lazy reading. This allows these libraries to read even faster than ffjson</code> and easyjson</code>.</p> By not having to deserialize to a struct, these libraries can read directly from the byte buffer which allows values to be read much faster than other deserialization libraries.</p> Another advantage over "encoding/json"</code>, is that the api is much more ergonomic than unmarshalling the json into a map[string]interface{}</code> since no casting is necessary and walking to the specific field you want to read is much easier. For example, here is how to read values using "encoding/json"</code> with map[string]interface{}</code> vs gjson</code>.</p> // "encoding/json" </span>package </span>main </span> </span>import </span>( </span> </span>"encoding/json" </span>) </span> </span>const </span>data </span>= </span>`{"name":{"first":"Janet","last":"Prichard"},"age":47}` </span> </span>func </span>main</span>() { </span> </span>var </span>values </span>map</span>[</span>string</span>]</span>interface</span>{} </span> json.Unmarshal([]</span>byte</span>(data), </span>&</span>values) </span> value </span>:= </span>values[</span>"name"</span>].(</span>map</span>[</span>string</span>]</span>interface</span>{})[</span>"last"</span>].(</span>string</span>) </span> printf(value) </span>} </span> </span>// gjson </span>package </span>main </span> </span>import </span>"github.com/tidwall/gjson" </span> </span>const </span>json </span>= </span>`{"name":{"first":"Janet","last":"Prichard"},"age":47}` </span> </span>func </span>main</span>() { </span> value </span>:= </span>gjson.Get(json, </span>"name.last"</span>) </span> </span>println</span>(value.String()) </span>} </span></code></pre> The disadvantage of these libraries is that they only provide deserialization and not serialization. For fast serialization, you can rely on the structured data serializers.</p> Utilities and other helpful libraries</h2> Convert JSON to a Go Struct</h3> Instead of doing this job by hand, here are some utilities that can help you:</p> Online: https://mholt.github.io/json-to-go/</li> Locally: https://github.com/ChimeraCoder/gojson</li> </ul> Summary</h2> Useful JSON libraries: For generating encoders and decoders:</p> ffjson</a></li> easyjson</a></li> </ul> For reading unstructured json at a specific path:</p> jsonparser</a></li> gjson</a></li> </ul> Spice up your LaTeX with Font Awesome Unknown — Tue, 02 Feb 2016 00:00:00 +0000 LaTeX allows you to create amazing documents which are beautifully typeset. It gives you access to a wide breath of tools and graphics to improve your documents. One of these are the many already available symbols in LaTeX. Here is a comprehensive list of available symbols: Link</a> PDF</a>. You can even find symbols by drawing them at this service</a>. These include mathematical symbols, phonetic symbols, currency symbols or symbols for various languages.</p> I found out recently that you can include Font Awesome</a> icons as symbols in your latex documents using this CTAN package</a>. This allows you to use some modern icons such as the Github or LinkedIn logos to spice up your documents such as a resume if you write it in LaTeX. If you have a full TeX live distribution such as MacTeX, it should be already included. I was able to use it by adding \usepackage{fontawesome}</code> to my document and using the lualatex</code> command to compile my document. I wasn't able figure out why xelatex</code> failed to compile my document. If you do let me know.</p> The final result:</p> </p> You can checkout the source here: Link</a></p> A simple protocol for a TCP chat server and client Unknown — Sun, 21 Jun 2015 00:00:00 +0000 Introduction</h2> I will be talking about some motivating factors for why you would want to use a lower level networking protocol such as TCP and how you could implement your own application protocol over it.</p> Some prerequisites</h3> a general understanding of networking</li> a Go distribution</li> </ul> Motivation</h2> Dealing with an HTTP API such as a RESTful web API is often the easier way to interact with your server. Applications that only need to consume and send commands to the server asynchronously can fit this use case. But some applications such as a chat client or a game server need a 2-way method of communication between the client and server. This is accomplished by using a lower level protocol such as TCP or UDP.</p> We can think of TCP and UDP as a pipe between 2 endpoints and at each end of the connection we have a stream of bytes that can be read. I will note here that TCP is in this analogy a reliable pipe whereas we can consider UDP as a faster but unreliable pipe between the endpoints. Further detail about the differences of TCP and UDP are beyond the scope of this post, but I encourage you to do some research to learn more since networking has some very leaky abstractions nad understanding the layer underneath is always useful in debugging issues such as outages, latency and any properties you would like to have.</p> How can we use tcp</h2> Now that we know what we can use to have a continuous stream of data to communicate between 2 endpoints, how can we actually used that stream of bytes? We first need a way to interpret that stream of bytes so that we can read the messages passed on there.</p> There are several ways to do it:</p> We could use a set number of bytes</li> We could use sentinel values to mark the beginning and/or end of a message</li> Something else(let's come back to this)</li> </ul> The pros and Cons</h3> Pros of using a set number of bytes:</p> Size of buffer to allocate for the message does not change and is known in advance</li> Data in the message can be arbitrary and does not need to be escaped</li> </ul> Cons of using a set number of bytes:</p> Can be wasteful. What if we need to send 8 bytes but we are using 1024 byte buffers</li> Hard to extend. Once we declare the protocol we cannot change the number of bytes in messages</li> </ul> Pros of using sentinel values</p> Easy to implement. Read until we see the sentinel value.</li> Flexible in how much data we can send.</li> </ul> Cons of using sentinel values</p> We cannot use the sentinel values in our data. We need to carefully escape the data to be sent.</li> </ul> Is there a better option?</h2> Glad you asked.</p> We could define a protocol that specifies how much data to be sent by using the set number of bytes scheme. This essentially ends up being a header. We could define a protocol that sets the number of bytes for the header which always needs to be sent followed by the message body.</p> A simple example of using this scheme would be to have a 4 byte message for a 32bit int which would describe the length of the message followed by the message. This will allow us to allocate a buffer of that size and read until we receive all the bytes promised in the header.</p> This negates the wastefulness of the fixed number of bytes scheme, but it still makes it hard to extend this protocol without breaking backwards compatibility. For example, in our simple example protocol, we can send a maximum of 4GB in our messages. (4 bytes = 48 = 32 bits; 2^32-1 = 4294967295 bytes = 4.29497 GB)</p> Lets use this to implement a simple chat server and client</h2> Using this scheme we can wrap the net.Conn</code> Read</code> method in a helper function as follows:</p> func </span>ReadMsg</span>(</span>conn </span>net.</span>Conn</span>) (</span>string</span>, </span>error</span>) { </span> </span>// Make a buffer to hold length of data </span> lenBuf </span>:= </span>make</span>([]</span>byte</span>, </span>4</span>) </span> _, err </span>:= </span>conn.Read(lenBuf) </span> </span>// Receiving EOF means that the connection has been closed </span> </span>if </span>err </span>== </span>io.EOF { </span> </span>// Close conn and exit </span> conn.Close() </span> fmt.Println(</span>"Connection Closed. Bye bye."</span>) </span> os.Exit(</span>0</span>) </span> } </span> </span>if </span>err </span>!= </span>nil </span>{ </span> </span>return </span>""</span>, err </span> } </span> lenData </span>:= </span>FromBytes(lenBuf) </span> </span> </span>// Make a buffer to hold incoming data. </span> buf </span>:= </span>make</span>([]</span>byte</span>, lenData) </span> reqLen </span>:= </span>0 </span> </span>// Keep reading data from the incoming connection into the buffer </span> </span>// until all the data promised is received </span> </span>for </span>reqLen </span>< </span>int</span>(lenData) { </span> tempreqLen, err </span>:= </span>conn.Read(buf[reqLen:]) </span> reqLen </span>+= </span>tempreqLen </span> </span>if </span>err </span>== </span>io.EOF { </span> </span>return </span>""</span>, fmt.Errorf(</span>"Received EOF before receiving all promised data."</span>) </span> } </span> </span>if </span>err </span>!= </span>nil </span>{ </span> </span>return </span>""</span>, fmt.Errorf(</span>"Error reading: </span>%s</span>"</span>, err.Error()) </span> } </span> } </span> </span>return </span>string</span>(buf), </span>nil </span>} </span></code></pre> What this does is read 4bytes from the specified connection and converts those bytes to an int. It then reads from the connection until it has seen as many bytes from the connection as was promised in the header. The FromBytes</code> function takes care of converting the bytes to the int. In our protocol, the int are encoded in binary in Big Endian as is common in most networking protocol which is why big endian is also known as The Network Byte Order</em>. Here is the helper functions to convert ints to and from bytes:</p> // To convert Big Endian binary format of a 4 byte integer to int32 </span>func </span>FromBytes</span>(</span>b </span>[]</span>byte</span>) </span>int32 </span>{ </span> buf </span>:= </span>bytes.NewReader(b) </span> </span>var </span>result </span>int32 </span> err </span>:= </span>binary.Read(buf, binary.BigEndian, </span>&</span>result) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> </span>return </span>result </span>} </span> </span>// To convert an int32 to a 4 byte Big Endian binary format </span>func </span>ToBytes</span>(</span>i </span>int32</span>) []</span>byte </span>{ </span> buf </span>:= </span>new</span>(bytes.</span>Buffer</span>) </span> err </span>:= </span>binary.Write(buf, binary.BigEndian, i) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> </span>return </span>buf.Bytes() </span>} </span></code></pre> Similarly for the write:</p> func </span>WriteMsg</span>(</span>conn </span>net.</span>Conn</span>, </span>msg </span>string</span>) { </span> </span>// Send the size of the message to be sent </span> conn.Write([]</span>byte</span>(ToBytes(</span>int32</span>(</span>len</span>([]</span>byte</span>(msg)))))) </span> </span>// Send the message </span> conn.Write([]</span>byte</span>(msg)) </span>} </span></code></pre> We write the size of the message and then the message.</p> Now lets use these in a server:</p> func </span>main</span>() { </span> </span>// Listen for incoming connections. </span> l, err </span>:= </span>net.Listen(CONN_TYPE, CONN_HOST</span>+</span>":"</span>+</span>CONN_PORT) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> fmt.Println(</span>"Error listening:"</span>, err.Error()) </span> os.Exit(</span>1</span>) </span> } </span> </span>// Close the listener when the application closes. </span> </span>defer </span>l.Close() </span> fmt.Println(</span>"Listening on " </span>+ </span>CONN_HOST </span>+ </span>":" </span>+ </span>CONN_PORT) </span> </span>for </span>{ </span> </span>// Listen for an incoming connection. </span> conn, err </span>:= </span>l.Accept() </span> </span>if </span>err </span>!= </span>nil </span>{ </span> fmt.Println(</span>"Error accepting: "</span>, err.Error()) </span> os.Exit(</span>1</span>) </span> } </span> </span>// Handle connections in a new goroutine. </span> </span>go </span>handleRequest(conn) </span> } </span>} </span> </span>// Handles incoming requests. </span>func </span>handleRequest</span>(</span>conn </span>net.</span>Conn</span>) { </span> </span>// Close the connection when you're done with it. </span> </span>defer </span>conn.Close() </span> </span> msg, err </span>:= </span>common.ReadMsg(conn) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> fmt.Printf(</span>"Message Received: </span>%s\n</span>"</span>, msg) </span> </span>// Send a response back to person contacting us. </span> common.WriteMsg(conn, </span>"Message Received."</span>) </span>} </span></code></pre> Here we setup the server to listen for connections at a given port and handle the connection when we receive one. You will notice we are closing the connection as soon as we send a reply, but this is not necessary and we could listen for several messages depending on your application.</p> On the client side:</p> func </span>main</span>() { </span> tcpAddr, err </span>:= </span>net.ResolveTCPAddr(</span>"tcp"</span>, CONN_HOST</span>+</span>":"</span>+</span>CONN_PORT) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> </span> </span>// Connect to server through tcp. </span> conn, err </span>:= </span>net.DialTCP(</span>"tcp"</span>, </span>nil</span>, tcpAddr) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> </span>defer </span>conn.Close() </span> </span>go </span>printOutput(conn) </span> writeInput(conn) </span>} </span> </span>func </span>writeInput</span>(</span>conn </span></span>net.</span>TCPConn</span>) { </span> fmt.Println(</span>"Enter text:"</span>) </span> </span>for </span>{ </span> </span>// Read from stdin. </span> reader </span>:= </span>bufio.NewReader(os.Stdin) </span> fmt.Print(</span>"> "</span>) </span> text, _ </span>:= </span>reader.ReadString(</span>'\n'</span>) </span> common.WriteMsg(conn, text) </span> } </span>} </span> </span>func </span>printOutput</span>(</span>conn </span></span>net.</span>TCPConn</span>) { </span> </span>for </span>{ </span> </span> msg, err </span>:= </span>common.ReadMsg(conn) </span> </span>if </span>err </span>!= </span>nil </span>{ </span> log.Fatal(err) </span> } </span> fmt.Println(msg) </span> } </span>} </span></code></pre> We first connect to the server and the same port our server is listening to and set a goroutine to print all messages we receive from the connection, if the connection is closed we exit. On another thread of execution we listen for input on stdin and write messages to the connection.</p> And Voila. We have a client chatting with our server.</p> Concluding remarks</h2> You will probably notice we are not handling the errors and simply exit on failure on any errors in this example. Don't do that in your code. I just wanted to keep my example simple here. In an actual program, we would return err in the function that has one and the top-level would handle them.</p> You can now extend this example to send more meaningful things that just strings. Interesting things such as XML or JSON or Protobuf.</p> The full example is available HERE</a></p> Experiments with Processing Unknown — Fri, 08 May 2015 00:00:00 +0000 Processing is a programming language targeted at electronic arts and visual design communities. It can serve as the foundation for electronic sketchbooks and aims to teach it's target communities programming in a visual context. At its core processing is a simple programming language with a very lightweight C-style syntax, similar to Java in which it is implemented.</p> There are now several implementations of Processing such as the one used on this page using Processing.js</a>. The original one is still the easiest one to use in my opinion but I haven't given a try to the others very thoroughly.</p> The context</h2> I am moving to a new apartment soon and in the process of buying new furniture I have to map out the dimensions of my apartment. I could do a simple paper sketch or simply draw the map in a paint program but my inner geek compels me to try out something new. I have tried several declarative visual programming languages but I haven't found any that I was comfortable with. It may simply be because I am just not made for visual design but I wanted to give this another chance.</p> I tried D3.js for data visualizations and a few graphing libraries in javascript but after trying Processing again, I can state that Processing seems to have the simplest semantics as well as being the most flexible in what it can do. It even has 3D visualizations!</p> Without much more ado here is the map of my apartment.</p> The Map:</h2> </iframe> PFont f; </span> </span>void setup() { </span> size(1000, 800); </span> f = createFont("Arial",16,true); </span>} </span> </span>void draw() { </span> textFont(f,12); </span> // start with the door </span> line(10, 300, 400, 300); </span> line(10, 300, 10, 310); </span> rect(10, 310, 3, 30); </span> line(10, 340, 10, 350); </span> text("Front Door",11,325); </span> line(10,350,300,350); </span> text("258x115",125,320); </span> </span> // kitchen </span> line(260,300, 260,80); </span> line(260,80, 400,80); </span> line(400,80, 400,300); </span> rect(315, 300, 30,3); </span> text("Kitchen", 300, 200); </span> text("170", 410,190); </span> </span> // kitchen column </span> line(400,275, 380,275); </span> line(380,300, 380,275); </span> </span> // bathroom </span> line(220,350, 220,500); </span> line(220,500, 300, 500); </span> line(300,500, 300,350); </span> rect(245,350, 30,3); </span> </span> // bedroom </span> line(300,350, 300,640); </span> line(300,640, 500,640); </span> line(500,640, 500,350); </span> text("Bedroom", 370, 470); </span> text("208",380,625); </span> text("287", 475,500); </span> </span> // living room </span> line(400,80, 835,80); </span> line(835,80, 835,350); </span> line(835,350, 500,350); </span> text("Living room", 550, 200); </span> text("435", 570, 95); </span> text("305", 805, 170); </span> text("142", 800, 345); </span> text("235",600,345); </span> </span> // tv stand </span> rect(720,350, 60,6); </span> text("TV Stand", 750, 370); </span> </span> // balcony door </span> rect(835,100, 3,30); </span> </span> // window </span> rect(835,180, 4,150); </span>} </span></code></pre> Moving from markdown to asciidoc Unknown — Sat, 11 Apr 2015 00:00:00 +0000 Markdown is probably the most popular markup language in the tech community. Often used in comment sections, blogs (such as jekyll), readmes on github or even documentation. Its popularity can easily be explained by its simplicity and extensibility, but this simplicity is both it's advantage and it's biggest weak point. Markdown is so simple that most of its users often extend it into a dialect and that's why there are so many versions of markdown (github markdown, stackoverflow markdown, etc...). If you are interested in markdown, it is worthwhile to check out CommonMark</a> which is a positive step in trying to standardize markdown and specify many of its extensions.</p> I have been using markdown for a few years and unless I am writing a long form document with rich formatting, I am often writing in markdown. It is very readable, easy and quick to write, and the fact that github renders markdown are the main reasons for why I use it, but I have been noticing some problems when it comes to writing slightly longer documents with structure such as notes. Breaking out LaTeX would not be very productive, so I found a solution: asciidoc</a>.</p> Asciidoc is also a readable document format which has equivalents for any of markdowns constructs and more. Things such as table of contents, continuous blocks can be grouped in elements, etc. It feels just as simple as markdown with much more power for richer formatting if needed. I have already converted some of my documents over to asciidoc and the process has gone swimmingly well.</p> Here is a simple readme in asciidoc:</p> = Readme </span>Author Name <name@email.com> </span> </span>== Heading 1 </span> </span>[source,haskell] </span>file.hs </span>---- </span>collatz 1 = [1] </span>collatz n </span> \| even n = n:chain (n `div` 2) </span> \| odd n = n:chain (3n + 1) </span>---- </span> </span>=== list of stuff: </span>* Lorem ipsum dolor sit amet, consectetur </span>* Donec lorem purus, vulputate et elementum tempus </span>* Ut blandit sagittis arcu, sit amet </span></code></pre> Some quick references while writing your asciidoc documents:</p> Asciidoc quick reference</a></li> [http://asciidoctor.org/docs/user-manual](Asciidoctor User Manual)</li> </ul> What I learned from Functional Programming Principles Unknown — Mon, 23 Feb 2015 00:00:00 +0000 Functional Programming Principles is a course taught at Ecole polytechnique fédérale de lausanne and on coursera by Prof. Martin Odersky.</p> I originally took the class to get familiar with scala before tackling the more interesting class by Martin 'Principles of Reactive Programming' which is about asynchronous computation and distributed computation. I had already had an intro to functional programming through SML in the COMP302 class taught at McGill by the brilliant Prof. Brigitte Pientka.</p> The class starts with a model of evaluation for functional programming that is a useful mental model of how to think about functional programs. It is interesting to note that SICP does a similar introduction to programming which I think is better than the usual hello world program many intro cs classes start with.</p> The rest of the class is about functional programming in scala and the various data structures available in the extensive scala standard library, such as lists, sets, maps and streams. The type hierarchy of the collections library is as follows:</p> +--Iterable </span> +--Seq </span> +--IndexedSeq </span> +--Vector </span> +--(Array) Imported from java does not subclass IndexSeq but </span> +--(String) Use wrappers to behave like IndexedSeq </span> +--LinearSeq </span> +--List </span> +--Set </span> +--Map </span></code></pre> Martin also when through some inductive proofs on properties of lists and trees. He also showed how for-expressions can be mapped to map and flatMap and can be more readable in many situations. The course ended with how to operate on lazy collections such as streams.</p> I liked the course overall as it was a good refresher on functional programming as well and a way to gain familiarity to Scala which is probably a factor for why Scala is gaining popularity in some circles. Scala is an interesting language which mixes object oriented programming with functional programming in a way that feels very natural.</p> You can see my notes on the course at : Link</a></p> Some notes on bash scripting Unknown — Fri, 20 Feb 2015 00:00:00 +0000 In your day to day tasks you often encounter tasks that can be easily automated. Instead of wasting time doing menial or mechanical tasks, you could be applying your brain power to automating them. If you are a developer, this is doubly relevant to you to manage your processes, build, cleaning your project directory, etc.</p> Tasks such as massaging data from one format to another or quickly searching and replacing words in certain files, can easily be dealt with a scripting language. One scripting language that is widely available on most modern Linux and Unix systems is bash. Many people will argue to write in sh which is described in the POSIX standard, but we won't be going into this debate here. Instead I can state that most of these tips will be relevant for both.</p> Why not Python, Perl or Ruby?</h2> Although I believe these more fully featured language are appropriate for many tasks even beyond simple scripts, I think it is still a useful skill to learn to script with your shell.</p> I have been doing some devops works recently and had to use a lot of shell scripting to glue together the continuous integration pipeline. I don't claim to be a bash expert but here are some tips I picked up.</p> Don't hardcode the path to bash in your shebang</h2> Instead use #!/usr/bin/env bash</code>. This allows your script to be more portable since bash is not always located under /bin/bash in all linux distros.</p> Always use set -e</h2> When running most scripts that are not simple on off jobs, you may want to be notified of an error occurring and stop the execution of the script. By using "set -e" at the top of your script, it will prevent the execution of the rest of your job if something fails. This allows you to be explicit when you can allow some command to fail which is rarer than when you expect all the commands to run.</p> set </span>-e </span></code></pre> Use set -x for better debugging and logs</h2> When running a script whose output will be captured such as in a ci pipeline, it is useful to know what command was running last when something occurred, such as when an exception occurred. Pretty useful for logs for this same reason.</p> set </span>-x </span></code></pre> Use getopts for easy command line options</h2> Instead of parsing the command line parameters manually you can simply use the getopts library to do the job. It supports only short command line options to make things simpler.</p> #!/usr/bin/env bash </span> </span>while </span>getopts </span>"h:f"</span> OPTION</span>; do </span> </span>case </span>$OPTION </span>in </span> h</span>) </span> </span>echo </span>"usage" </span> ;; </span> f</span>) </span> FILE</span>=</span>$OPTARG </span> </span>echo </span>$FILE </span> ;; </span> </span>esac </span>done </span></code></pre> A column ':' before a character tells getopts which character needs a value. A column ':' at the beginning of the optstring (e.g. ":h:s") sets OPTION var with “?” and the $OPTARG with the wrong character and no output to stderr. Otherwise it would print an illegal argument message to stderr, if an invalid flag is used.</p> Print a multiline string</h2> Instead of printing with several echos for each line use here documents.</p> cat </span><< EOF </span>usage: dotfiles [-h][-u][-p][-d] </span> </span>syncs dotfiles between ~/dotfiles and ~/ </span> </span>OPTIONS: </span> -h print usage </span> -u updates dotfiles in ~/ </span> -p pushes changes to dotfiles in ~/ to ~/dotfiles </span> -d view the difference between dotfiles in ~/ and ~/dotfiles </span>EOF </span></code></pre> Lexing and parsing in Go (Part 1) Unknown — Wed, 04 Feb 2015 00:00:00 +0000 Note: all the code can be seen at github.com/jonfk/calc</a></em></p> About a month ago, I watched a video of Rob Pike</a> giving a talk about lexing for the text/template/parse package. In it he described the simplicity and elegance of making a traditionally not concurrent problem into a concurrent solution. This then motivated me to try out making a lexer for a yet undefined language of my own design. But first as I did in my implementation, let start with the lexer.</p> Lexical Analysis</h2> My quest started simply with a lexer. The idea behind the lexer is actually quite simple. The lexer has a goroutine that runs the lexer and another that collects the tokens sent by the lexer. This setup allows the lexer to be run in a very simple manner. Instead of having mutually recursive functions, we have a for loop that runs functions set by the lexer. The for loop is as follows:</p> // stateFn represents the state of the scanner as a function </span>// that returns the next state. </span>type </span>stateFn </span>func</span>(</span></span>Lexer</span>) </span>stateFn </span> </span>// run runs the state machine for the lexer. </span>func </span>(</span>l </span></span>Lexer</span>) </span>run</span>() { </span> </span>for </span>l.state </span>= </span>lexStart; l.state </span>!= </span>nil</span>; { </span> l.state </span>= </span>l.state(l) </span> } </span>} </span></code></pre> As you will notice, l is a pointer to the lexer struct and lexStart is the first function passed to the loop. The state field of the lexer holds a state function which is a function with the above type signature. The state function returns another state function which will take care of the next state and so on.The for loop then runs the current state until a state function returns nil. The run() function is ran in a goroutine and communicates with the main goroutine through a channel</p> // lex creates a new scanner for the input string. </span>func </span>Lex</span>(</span>name</span>, </span>input </span>string</span>) </span></span>Lexer </span>{ </span> l </span>:= &</span>Lexer{ </span> name: name, </span> input: input, </span> items: </span>make</span>(</span>chan Token</span>), </span> } </span> </span>go </span>l.run() </span> </span>return </span>l </span>} </span></code></pre> In the other goroutine, users of the lexer can make calls to NextItem() which returns the next Token in the channel.</p> Tokens</h2> At this point, you would be right to wonder what exactly are we lexing, since we cannot lex tokens we don't know. I started with the code from the text/template/parse package</a> and modified it to suit my needs. Tokens are defined as a struct with fields for the type, position and value of the token.</p> // token represents a token or text string returned from the scanner. </span>type </span>Token </span>struct </span>{ </span> Typ </span>TokenType </span>// The type of this token. </span> Pos </span>Pos </span>// The starting position, in bytes, of this item in the input string. </span> Val </span>string </span>// The value of this item. </span>} </span></code></pre> The TokenType is then an int defined with the const iota specs. e.g.</p> const </span>( </span> ERROR </span>TokenType </span>= </span>iota </span>// error occurred; value is text of error </span> BOOL </span>// boolean constant </span> EOF </span> NEWLINE </span>// '\n' </span> </span> </span>... </span> </span> </span>// Keywords appear after all the rest. </span> KEYWORD </span>// used only to delimit the keywords </span> ELSE </span>// else keyword </span> END </span>// end keyword </span> IF </span>// if keyword </span> THEN </span>// then keyword </span> LET </span>// let keyword </span> VAR </span>// var keyword </span> VAL </span>// val keyword </span> </span> OPERATOR </span> </span>// Operators and delimiters </span> ADD </span>// + </span> SUB </span>// - </span> MUL </span>// </span> QUO </span>// / </span> REM </span>// % </span> </span> </span>... </span>) </span></code></pre> The TokenType is mapped to their string representation using a map.</p> var </span>key </span>= </span>map</span>[</span>string</span>]</span>TokenType</span>{ </span> </span>"else"</span>: ELSE, </span> </span>"end"</span>: END, </span> </span>"if"</span>: IF, </span> </span>"then"</span>: THEN, </span> </span>"let"</span>: LET, </span> </span>"val"</span>: VAL, </span> </span>"var"</span>: VAR, </span> </span>"+"</span>: ADD, </span> </span>"-"</span>: SUB, </span> </span>"*"</span>: MUL, </span> </span>"/"</span>: QUO, </span> </span>"%"</span>: REM, </span> </span>"&&"</span>: LAND, </span> </span>"\|\|"</span>: LOR, </span> </span>"=="</span>: EQL, </span> </span>"<"</span>: LSS, </span> </span>">"</span>: GTR, </span> </span>"="</span>: ASSIGN, </span> </span>"!"</span>: NOT, </span> </span>"!="</span>: NEQ, </span> </span>"<="</span>: LEQ, </span> </span>">="</span>: GEQ, </span>} </span></code></pre> A few other helper functions and methods are defined such as precedence(), IsLiteral(), Equals(), etc.</p> Next we will talk about the ast.</p> How I setup my emacs config Unknown — Sun, 21 Dec 2014 00:00:00 +0000 Last Edited 04 Feb 2015</em></p> First some background. I used vim for about 2 years. It was my first real editor back when I was a first year CompSci student. It was a hard road in the beginning, learning modal editing, the quirks of vim's configuration and various useful plugins using Tim Pope's pathogen</a>.</p> After a few years of vim usage, you could wonder why I switched, especially since I go out of my way to set up emacs to replicate how I was using vim. Extensibility is the main reason. In emacs, the ease of extending the capabilities of your editor to suit your needs makes my switch worthwhile. I haven't given a real try to elisp yet (that's on my todo list) but binding a function to a keymap is much easier to do and extending the functionality of your editor through it's API is a breeze.</p> One of the well known sayings about emacs is : "a great operating system, lacking only a decent editor". So although the default editor that comes with emacs is not great, you can make it your</i> best editor. The following packages are what makes emacs a better editor for me.</p> EVIL</h2> The most important piece of my emacs config is evil-mode. Evil brings modal editing to emacs and replicates many of vim's features. All the keybindings I was used to in vim are there. I don't claim to be the most advanced vim user out there. There are many who use vim's advanced features in such a way that would prevent them from switching, but if your main reason to stay with vim is modal editing, I urge you to give evil a try.</p> package.el</h2> Another reason why I prefer emacs is that since emacs24, package.el comes on the default installation. This means that I can bootstrap my emacs from it's config file and have my customized environment in a matter of seconds.</p> Evil Leader</h2> Although not as essential as it's namesake mode, evil-leader is useful to prevent rsi and map many of emacs obscure keybindings to a simple 2 keypress. For example I map my leader to the ,</code> key and kill-buffer</code> is mapped to k, so to kill the current buffer I do ,-k</code></p> My config is linked here: .emacs</a></p>

Bye tmux, hello neovim. How I am trying neovim as my terminal multiplexer

My Tmux config and workflow</h1>
You can find my full unedited tmux config at this link: 70e3965</a>.</p>

A Git Configuration Overhaul

`Sources</h2> How Core Git Developers Configure Git</a></li> HN comments for "How Core Git Developers Configure Git"</a></li> Fixing up Git with Autosquash</a></li> </ul>`

My keyboard symbol layer

NFS vs SMB transfer speed on a Linux server

Setup</h1>

Network</h2>
The NAS and server are connected wired through a 2.5GbE switch.</p>

Results</h1>
For the raw results see fio_output.txt</a>. Otherwise here a summary created using the analysis.py</a> script follows in the next section.</p>

My Personal Self-Hosted Infrastructure Part 1

Conclusion</h2>
Self-hosting offers a powerful way to take control of your digital life, but it requires careful planning and a willingness to learn. As my journey shows, starting small and gradually expanding can lead to a robust and personalized infrastructure that meets your specific needs.</p>

RFC Processes for Software Development

Trying to understand how Next.js render server components on updates

Playing with Automerge and text CRDTs

Integrating Ory Hydra OAuth 2.0 with a Rust web service

Integrating with Hydra</h3>
To integrate with the Hydra OAuth2 server, we need to implement a login and consent application. You can review the documentation for doing so here</a>.</p>

Go and JSON, various useful libraries and utilities

Utilities and other helpful libraries</h2>

Summary</h2>
Useful JSON libraries: For generating encoders and decoders:</p>

ffjson</a></li>
easyjson</a></li> </ul>
For reading unstructured json at a specific path:</p>

jsonparser</a></li>
gjson</a></li> </ul>

Spice up your LaTeX with Font Awesome

A simple protocol for a TCP chat server and client

Introduction</h2>
I will be talking about some motivating factors for why you would want to use a lower level networking protocol such as TCP and how you could implement your own application protocol over it.</p>

Experiments with Processing

Moving from markdown to asciidoc

What I learned from Functional Programming Principles

Some notes on bash scripting

Don't hardcode the path to bash in your shebang</h2>
Instead use `#!/usr/bin/env bash</code>. This allows your script to be more portable since bash is not always located under /bin/bash in all linux distros.</p>`

Lexing and parsing in Go (Part 1)

How I setup my emacs config

package.el</h2>
Another reason why I prefer emacs is that since emacs24, package.el comes on the default installation. This means that I can bootstrap my emacs from it's config file and have my customized environment in a matter of seconds.</p>

Jonfk

Exploring Neovim Configs can be Tiring

My Configs So Far</h2>
I have been using kickstart.nvim</strong> for a while now and this rabbit hole began when I decided it was time to upgrade my own config</strong> and figured I’d see what other distros were doing right. I tried out LazyVim, LunarVim, NvChad, and friends.</p>

Jonfk

Exploring Neovim Configs can be Tiring

My Configs So Far</h2> I have been using kickstart.nvim</strong> for a while now and this rabbit hole began when I decided it was time to upgrade my own config</strong> and figured I’d see what other distros were doing right. I tried out LazyVim, LunarVim, NvChad, and friends.</p>

Bye tmux, hello neovim. How I am trying neovim as my terminal multiplexer

Vi Copy Mode</h2> For getting text in and out of a tmux buffer, I use the vi</code> copy-mode. It allows me to use vi keybindings to navigate, yank, paste, etc.</p>

Tmux windows in sessions and workflow</h2> A little quality of life thing I have is to start tmux windows at 1 instead of at 0. That's because it keeps all the window numbers close together even if you don't have a lot of windows. Not a super big thing, but helps a lot I find.</p>

A Git Configuration Overhaul

My keyboard symbol layer

NFS vs SMB transfer speed on a Linux server

Setup</h1>

Network</h2> The NAS and server are connected wired through a 2.5GbE switch.</p>

My Personal Self-Hosted Infrastructure Part 1

Conclusion</h2> Self-hosting offers a powerful way to take control of your digital life, but it requires careful planning and a willingness to learn. As my journey shows, starting small and gradually expanding can lead to a robust and personalized infrastructure that meets your specific needs.</p>

RFC Processes for Software Development

Trying to understand how Next.js render server components on updates

Playing with Automerge and text CRDTs

Integrating Ory Hydra OAuth 2.0 with a Rust web service

Hello World</h3> We can now use these dependencies to provide a basic web service. Let's write a basic hello world endpoint as in the example.</p>

Login and consent</h3> We showed the basic building blocks for creating a web service in the previous section. Let's put that to use and build our integration to our OAuth 2.0 server. We will implement the login and consent pages and their corresponding form post endpoints.</p>

Go and JSON, various useful libraries and utilities

Utilities and other helpful libraries</h2>

Convert JSON to a Go Struct</h3> Instead of doing this job by hand, here are some utilities that can help you:</p>

Summary</h2> Useful JSON libraries: For generating encoders and decoders:</p>

Spice up your LaTeX with Font Awesome

A simple protocol for a TCP chat server and client

Introduction</h2> I will be talking about some motivating factors for why you would want to use a lower level networking protocol such as TCP and how you could implement your own application protocol over it.</p>

Some prerequisites</h3> a general understanding of networking</li>

Experiments with Processing

Moving from markdown to asciidoc

What I learned from Functional Programming Principles

Some notes on bash scripting

Don't hardcode the path to bash in your shebang</h2> Instead use #!/usr/bin/env bash</code>. This allows your script to be more portable since bash is not always located under /bin/bash in all linux distros.</p>

Lexing and parsing in Go (Part 1)

How I setup my emacs config

package.el</h2> Another reason why I prefer emacs is that since emacs24, package.el comes on the default installation. This means that I can bootstrap my emacs from it's config file and have my customized environment in a matter of seconds.</p>

My Configs So Far</h2>
I have been using kickstart.nvim</strong> for a while now and this rabbit hole began when I decided it was time to upgrade my own config</strong> and figured I’d see what other distros were doing right. I tried out LazyVim, LunarVim, NvChad, and friends.</p>

Network</h2>
The NAS and server are connected wired through a 2.5GbE switch.</p>

Conclusion</h2>
Self-hosting offers a powerful way to take control of your digital life, but it requires careful planning and a willingness to learn. As my journey shows, starting small and gradually expanding can lead to a robust and personalized infrastructure that meets your specific needs.</p>

Summary</h2>
Useful JSON libraries: For generating encoders and decoders:</p>

ffjson</a></li>
easyjson</a></li> </ul>
For reading unstructured json at a specific path:</p>

jsonparser</a></li>
gjson</a></li> </ul>

Introduction</h2>
I will be talking about some motivating factors for why you would want to use a lower level networking protocol such as TCP and how you could implement your own application protocol over it.</p>

Don't hardcode the path to bash in your shebang</h2>
Instead use `#!/usr/bin/env bash</code>. This allows your script to be more portable since bash is not always located under /bin/bash in all linux distros.</p>`

package.el</h2>
Another reason why I prefer emacs is that since emacs24, package.el comes on the default installation. This means that I can bootstrap my emacs from it's config file and have my customized environment in a matter of seconds.</p>