index.json

[
 {
  "content": "Background India has seen significant growth in cafés serving specialty coffee in recent times. I recently visited one such place in Pune and had a great cup of fruity, floral coffee brewed using a vacuum siphon. This was my first experience of an Indian coffee with such a flavour profile. Until then, most of the Indian coffee beans I had tried were earthy or nutty in nature. There was this pronounced feeling that any Indian coffee with a medium to dark roast will taste very similar for a given brewing method.\nAfter this experience, I decided to understand the different coffees of India and how they get their flavour. Below is my understanding of the different types of coffee bean varieties available in India, their flavour profiles, and the associated processing that brings out these flavours in them.\nUnderstanding Coffee Flavours When we talk about coffee flavours, we’re referring to the complex blend of taste and aroma that defines each cup. These flavours range widely and are often misunderstood. It’s important to note that not all flavours are universally appealing. Individual preferences vary, and what one person loves, another might find less enjoyable.\nCommon coffee flavour descriptors:\nAcidity (brightness): The liveliness or sharpness of the coffee. It’s not about sourness but rather a crisp, tangy sensation similar to biting into a fresh apple. Body (mouthfeel): How heavy or light the coffee feels in your mouth. A coffee with a full body feels rich and heavy, like whole milk, while a light-bodied coffee feels more like water. Floral notes: Delicate, fragrant flavours reminiscent of flowers like jasmine or rose. Fruity notes: Tastes that evoke fruits, such as berries, citrus, tropical, or stone fruits, providing sweetness or tartness. Chocolatey notes: Flavours similar to cocoa or dark chocolate. Nutty notes: Subtle tastes like almonds, hazelnuts, or peanuts. Earthy notes: Flavours that remind you of soil or wood, often found in coffees from certain regions. Spicy notes: Hints of spices like cinnamon, clove, or pepper, often a result of inter-cropping. Bitterness: A sharp taste that can be pleasant in balance but overwhelming if too strong. Sweetness: A natural sweetness inherent in the coffee, not from added sugar. The Coffee Bean Lifecycle Growth and Harvesting Coffee plants are cultivated in suitable climates with the right soil conditions. The environment in which coffee is grown dramatically affects its flavour.\nIn India Coffee is primarily grown in the southern states of Karnataka, Kerala, and Tamil Nadu, in iron and aluminum-rich soils at elevations ranging from 700 to 1,600 meters. Karnataka produces about 70% of the country’s coffee. Regions like Baba Budangiri, Chikmagalur, and the Nilgiri Hills are renowned for high-quality Arabica. While the northeastern states also cultivate coffee, their main focus remains on tea. Shade-Grown Coffee: All coffee grown in India is shade-grown, commonly with two tiers of shade provided by different canopy trees. Shade-grown coffee tends to mature more slowly, allowing sugars to develop more fully. Shade-grown coffee plants are often more resilient to diseases and pests due to the balanced ecosystem. In contrast, sun-grown coffee often matures faster, leading to higher yields but can result in beans with less complex flavours. Inter-Cropping with Spices: Coffee plants are often inter-cropped with spices such as cardamom, cinnamon, clove, and nutmeg. The proximity of these spices can impart subtle aromatic notes to the coffee beans, adding complexity to the flavour profile. Potential Challenges: Pests and diseases like coffee leaf rust remain significant challenges for Arabica cultivation. Large efforts are dedicated to developing resistant bean types that are flavour-rich and can thrive in Indian soil conditions. Altitude and Its Impact High Altitude (Above 1,000 meters): Bean Development: Slower maturation allows complex sugars to develop. Beans are denser and harder. Flavour Impact: Higher acidity with bright, vibrant flavours. Notes of florals and fruits. Varieties Grown: Arabica beans, including varieties like Kent, Selection 9, Sln.795, Sln.5B/S13, and Sln.6. Low Altitude (Below 1,000 meters): Bean Development: Faster maturation. Beans are less dense. Flavour Impact: Lower acidity with fuller body. Earthy and nutty flavours. Varieties Grown: Robusta beans, such as S.274. Harvesting Harvesting involves picking ripe coffee cherries from the plants. Picking Methods: Selective Picking: Only the ripe cherries are picked individually by hand. Ensures optimal ripeness and higher quality. Labour-intensive and time-consuming. Strip Picking: All cherries, regardless of ripeness, are stripped from the branch at once. Faster and less labour-intensive. Results in a mix of ripe and unripe cherries, leading to inconsistent flavours. Potential Issues: Overripe or Underripe Cherries: Picking cherries at the wrong time can lead to inconsistent flavours, with overripe cherries contributing to fermented tastes and underripe cherries adding sourness. Quality Control: Not carefully selecting ripe cherries can compromise the overall quality of the batch. Processing Processing removes the coffee beans from the cherries and prepares them for drying. These methods significantly impact the coffee’s final flavour profile.\n1. Washed Process (Wet Process) Method: Removal of skin and pulp using water. Fermentation to remove mucilage. Beans are washed and then dried. Flavour Impact: Clean, bright flavours with pronounced acidity. Highlights the inherent characteristics of the bean. Potential Issues: Over-fermentation can cause sour flavours. Requires significant water resources. 2. Natural Process (Dry Process) Method: Whole cherries are dried under the sun. Once dried, the beans are hulled to remove the fruit. Flavour Impact: Sweet, fruity flavours with a heavier body. Adds complexity and depth. Potential Issues: Risk of inconsistent drying leading to mould. May develop off-flavours if not properly managed. 3. Honey Process (Pulped Natural) Method: Skin is removed, but some or all of the mucilage is left during drying. Variations include yellow, red, and black honey, depending on the amount of mucilage. Flavour Impact: Balanced acidity and sweetness. Syrupy mouthfeel. Potential Issues: Requires careful monitoring to prevent fermentation defects. 4. Anaerobic Fermentation Method: Coffee cherries or pulped beans are fermented in sealed tanks without oxygen. Environment encourages specific microorganisms that influence flavour. Flavour Impact: Intense, unique flavours with enhanced fruitiness and sweetness. Complex and sometimes exotic taste profiles. Potential Issues: Risk of developing undesirable flavours if not precisely controlled. Requires expertise and careful monitoring. 5. Carbonic Maceration Method: Whole coffee cherries are placed in sealed tanks filled with carbon dioxide. Mimics winemaking techniques. Flavour Impact: Vibrant, fruity notes with bright acidity. Can produce flavours not typically associated with coffee. Potential Issues: Complex process needing precise control over variables. Can be expensive and time-consuming. 6. Natural Maceration Method: Similar to natural processing but with extended fermentation. Cherries are fermented in open tanks or barrels before drying. Flavour Impact: Deep, wine-like flavours with rich fruitiness. Enhanced sweetness and body. Potential Issues: Improper Fermentation: High risk of over-fermentation and spoilage, leading to off-flavours like sourness or alcohol-like tastes. Contamination: Mould or bacterial growth during processing can ruin the beans. 7. Monsooning (Unique to India) Method: Monsooned Malabar: Green beans are exposed to monsoon winds and moisture in open warehouses for 12–16 weeks. Beans absorb moisture, swell, and change colour to pale gold. Flavour Impact: Low acidity with heavy body and a practically neutral pH balance. Unique earthy, musty, woody, and chocolatey aroma with notes of spice and nut. Smooth and mellow cup. Potential Issues: Musty flavours may not appeal to everyone. Requires specific climatic conditions and careful handling. Drying and Milling After processing, beans are dried and milled to remove the outer layers.\nPotential Issues: Uneven Drying: Can lead to mould growth or fermentation defects. Overdrying or Underdrying: Affects bean quality and shelf life. Roasting Roasting transforms green coffee beans into the aromatic brown beans we recognize. Roast levels affect not just flavour but also physical properties, impacting brewing techniques.\nDensity Changes: Lighter Roasts: Beans are smaller, denser, and retain more moisture. Darker Roasts: Beans are larger, less dense, lighter in weight, with very little moisture. Impact on Brewing: Grind Size: Lighter roasts may require a finer grind due to higher density. Dosing: Dark roasts occupy more space; the same weight may fill your portafilter differently. Oil Migration: Light Roast: Minimal oils on the surface. Dark Roast: Oils migrate to the surface, giving a shiny appearance. Considerations Freshness: Surface oils can oxidize, leading to stale flavours. Equipment Maintenance: Oily beans may clog grinders and brewing equipment. Potential Issues: Over-Roasting: Results in burnt, bitter flavours. Under-Roasting: Leads to grassy, underdeveloped flavours. Grinding and Brewing The final stage involves grinding the roasted beans and brewing the coffee. This is a very large topic in itself, and I won’t go into a lot of details of it.\nPotential Issues: Incorrect Grind Size: Can cause over-extraction (bitterness) or under-extraction (sourness). Improper Brewing Techniques: Temperature and time affect flavour extraction. Indian Coffee Bean Varieties India cultivates several species of coffee, each offering distinct characteristics and flavours. The primary species grown are Arabica, Robusta, and, to a lesser extent, Liberica. Understanding the differences among these beans is essential for appreciating the diversity of Indian coffee.\nArabica Beans (Coffea arabica) Description: Arabica is the most widely consumed coffee species globally, known for its delicate flavour and lower caffeine content compared to Robusta. Thrives at higher altitudes between 900 and 2,000 meters. Characteristics: Taste: Mild acidity, complex flavours ranging from fruity to floral. Caffeine Content: Around 1.2% caffeine by weight. Bean Shape: Oval with a pronounced center crease. India is renowned for its high-quality Arabica beans. Let’s explore some of the prominent varieties.\n1. Kent Description: One of India’s oldest and most esteemed Arabica varieties, developed in the 1920s. Initially resistant to coffee leaf rust. Heritage: Selected from Typica plants. Growing Regions: Baba Budangiri, Chikmagalur, Coorg, Nilgiri Hills. Altitude: 1,000–1,600 meters. Flavour Profile: Mild acidity with floral aromas. Notes of citrus and a balanced body. Roasting Considerations: Best suited to light to medium roasts to preserve delicate flavours. Usage: Ideal for filter brewing methods to highlight nuanced flavours. 2. Sln.795 (Selection 795) Description: Developed in the 1940s by crossing Kent with S.288, a hybrid of Arabica and Liberica. High-yielding variety with rust resistance and good cup quality. Heritage: Cross between Kent and S.288. Growing Regions: Widely cultivated across South India. Altitude: 700–1,500 meters. Flavour Profile: Good acidity with notes of chocolate, caramel, and hints of spice. Medium to full body. Roasting Considerations: Versatile across roast levels. Usage: Suitable for espresso and milk-based drinks. 3. Cauvery (Catimor) Description: Also known as Catimor; developed as a rust-resistant variety. Heritage: A cross between Caturra (a mutation of Bourbon) and Hybrido-de-Timor. Growing Regions: Karnataka, Tamil Nadu. Altitude: 600–1,200 meters. Flavour Profile: Moderate acidity with sweet notes. Flavours of berries, chocolate, and sometimes a hint of spice. Roasting Considerations: Medium roasts highlight its balance of body and flavour. Usage: Good for both espresso and filter coffee. 4. Selection 9 (Sln.9) Description: Award-winning variety with excellent cup quality. Winner of the Fine Cup Award for best Arabica at the 2002 Flavour of India – Cupping Competition. Heritage: Hybrid of Tafarikela (an Ethiopian Arabica variety) and Hybrido-de-Timor. Growing Regions: Chikmagalur, Coorg. Altitude: 1,000–1,600 meters. Flavour Profile: Bright acidity with floral and fruity notes. Hints of jasmine and citrus. Roasting Considerations: Light roasts best showcase its vibrant flavours. Usage: Perfect for pour-over and specialty brewing methods. 5. Sln.5B (S13) Description: Hybrid variety developed for disease resistance and quality. Heritage: Cross between S.288 and Kent. Growing Regions: Karnataka, Tamil Nadu. Altitude: 800–1,500 meters. Flavour Profile: Balanced acidity with sweet notes. Flavours of berries and milk chocolate. Roasting Considerations: Light to medium roasts highlight its sweetness. Usage: Suitable for both espresso and filter coffee. 6. Sln.6 Description: Developed for improved yield, rust resistance, and quality. Heritage: Cross between S.288 and S.26. Growing Regions: Kerala, Karnataka. Altitude: 900–1,600 meters. Flavour Profile: Mild acidity with nutty and spice notes. Smooth body. Roasting Considerations: Medium roasts enhance its body and flavour. Usage: Versatile; suitable for various brewing methods. 7. Mysore Nuggets Extra Bold Description: A premium grade of Arabica beans from Karnataka. Beans are extra-large (screen size 19), uniform, and known for their superior quality. Growing Regions: Chikmagalur, Bababudangiri, Coorg. Altitude: 1,200–1,600 meters. Flavour Profile: Rich aroma with moderate acidity. Notes of chocolate, sweet-toned spicy flavours, and a hint of herbs. Roasting Considerations: Medium to dark roasts develop its complexity. Usage: Excellent for espresso and drip coffee. Robusta Beans (Coffea canephora) Description: Robusta beans are known for their hardiness and ability to grow at lower altitudes and higher temperatures. Often used in blends to add body and crema, especially in espressos. Characteristics: Taste: Strong, full-bodied with earthy flavours and higher bitterness. Caffeine Content: Around 2.2% caffeine by weight, higher than Arabica. Bean Shape: Rounder and smaller than Arabica beans. India produces some high-quality Robusta beans. Here are the notable varieties.\n1. S.274 Description: Popular Indian Robusta variety known for its hardiness and high yield. Growing Regions: Coorg, Wayanad, Chikmagalur. Altitude: Up to 1,000 meters. Flavour Profile: Full body with earthy tones. Notes of dark chocolate and spice. Roasting Considerations: Handles dark roasts well. Usage: Common in espresso blends and South Indian filter coffee. 2. Robusta Kaapi Royale Description: Premium grade Robusta beans; “Kaapi” is South Indian for coffee. Beans are large, even-sized, and of superior quality. Growing Regions: Similar regions as S.274. Altitude: Up to 1,000 meters. Flavour Profile: Smooth with lower acidity. Hints of chocolate, nuts, and a lingering aftertaste. Roasting Considerations: Dark roasts bring out depth, but care is needed to avoid excessive bitterness. Usage: Enhances crema in espresso; used in high-quality blends. Liberica Beans (Coffea liberica) Description: Liberica beans are larger than Arabica and Robusta beans and have a unique, bold flavour profile. Grown in limited quantities in India, primarily in certain regions of Kerala. Characteristics: Taste: Distinct woody and smoky notes with a full body. Caffeine Content: Similar to Arabica. Bean Shape: Large and irregular. Growing Regions: Parts of Kerala and Tamil Nadu. Varieties: Specific varieties are less defined due to limited cultivation. Flavour Profile: Unique flavour with woody, smoky notes. Full body with hints of floral and fruity undertones. Usage: Often used in blends to add complexity. Due to its strong flavour, it is sometimes blended with Arabica and Robusta beans. Excelsa Beans (Coffea liberica var. dewevrei) Description: Once considered a separate species, Excelsa is now classified as a variety of Liberica. Grown very limitedly in India. Characteristics: Taste: Tart and fruity notes with a complex profile. Bean Shape: Similar to Liberica but smaller. Growing Regions: Rare in India; if available, it’s usually from experimental or specialty farms. Usage: Used in specialty blends to add depth and complexity. Chicory Additions Chicory is a significant component of traditional South Indian coffee culture.\nDescription: Chicory is a perennial plant whose roots are roasted, ground, and used as a coffee additive or substitute. It extends the coffee, adds body, and introduces unique flavour characteristics. Why Add Chicory? Extends the coffee supply and adds depth to the flavour. Contributes to a thicker mouthfeel and a slight bitterness. Growing Regions: In India, chicory is cultivated in regions like Uttar Pradesh and Gujarat. Flavour Profile: Roasted and nutty flavours similar to dark toast or roasted nuts. Adds slight bitterness that’s earthy and herbal. Subtle sweetness that balances the bitter notes. Usage: Blended with coffee in proportions ranging from 10% to 30% chicory. Enhances body and adds a unique flavour to the coffee. Roasting Considerations: Chicory roots are roasted until dark to develop deep flavours. Brewing Considerations: Blends containing chicory are often brewed using the traditional South Indian filter method. Typically enjoyed with milk and sweeteners to balance the robust flavours. Practical Recommendations To help you navigate the diverse world of Indian coffee, here are some practical tips based on your brewing preferences and desired flavours.\nFor South Indian Filter Coffee Lovers Recommended Beans: Blend of Arabica and Robusta: Combine Sln.795 (Arabica) with S.274 or Robusta Kaapi Royale (Robusta). Blend with chicory, with coffee constituting 80–90% and chicory 10–20%. The chicory’s slight bitterness and body contribute to the traditional flavour. Traditionally, jaggery or honey were used as sweeteners, but white sugar has been used since the mid-1900s. Roast Level: Medium to dark roast enhances body and richness. Expected Flavours: Bold, full-bodied cup with notes of chocolate and spice. Additional Tips: Use freshly ground coffee suitable for the South Indian filter. Use a traditional South Indian filter for an authentic experience. For Espresso Enthusiasts Recommended Beans: Sln.795 or Sln.6 for balanced flavours. Add Robusta Kaapi Royale for crema and intensity. Monsooned Malabar for a unique and bold flavour. Roast Level: Medium to dark roast for depth and balance. Medium roast for Monsooned Malabar to maintain balance between body and flavour. Expected Flavours: Rich crema with notes of chocolate, nuts, and a smooth finish. Additional Tips: Ensure a fine grind and proper tamping for optimal extraction. The distinct flavour of Monsooned Malabar or Robusta Kaapi Royale may be an acquired taste; try blending with other beans to adjust intensity. For Those Seeking Fruity and Floral Notes Recommended Beans: Selection 9 (Sln.9) or Kent or Sln.13. Processing Method: Look for beans processed using Natural, Carbonic Maceration, or Anaerobic Fermentation methods. Roast Level: Light to medium roast to preserve delicate flavours. Expected Flavours: Bright acidity with flavours of citrus, berries, and floral aromas. Brewing Methods: Ideal for pour-over, AeroPress, or siphon brewing. For Smooth and Balanced Cups Recommended Beans: Cauvery (Catimor) or Sln.6. Roast Level: Medium roast for balance. Expected Flavours: Mild acidity, hints of chocolate and nuts, and a smooth body. Brewing Methods: Drip coffee makers, Chemex, or siphon. General Tips Freshness is Key: Use freshly roasted beans and grind just before brewing. Water Quality Matters: Use filtered water to prevent off-flavours. Adjust Brewing Variables: Modify grind size, water temperature, and brew time to taste. Comparison with Coffees from Other Regions While Indian coffees have unique qualities, there are aspects that some coffee enthusiasts might find less appealing compared to coffees from other regions:\nAcidity Levels:\nIndian Coffees: Generally have lower acidity compared to African or Central American coffees. Impact: Might be perceived as less vibrant or lacking brightness. Flavour Complexity:\nIndian Coffees: Often present earthy, spicy notes with full body. Impact: May lack the fruity or floral complexity found in Ethiopian or Colombian coffees. Processing Methods:\nMonsooned Coffees: Unique to India, offering distinctive flavours. Impact: The musty, woody notes of monsooned coffees might not appeal to those preferring cleaner flavour profiles. Roasting Preferences:\nIndia: Tendency towards dark roasts, resulting in bolder, sometimes bitter flavours. Impact: Can overshadow the intrinsic characteristics of the beans. Use in Blends:\nIndian Robusta is often used in espresso blends for crema and body. Impact: On its own, Robusta’s higher bitterness and lower acidity may be less desirable for some. Examples:\nSouth Indian Filter Coffee: Uses dark-roasted coffee, often blended with chicory. Might be too strong or bitter for those accustomed to lighter roasts. Monsooned Malabar: While prized for its unique profile, the earthy and musty flavours can be off-putting to some. Final Thoughts By embracing the unique characteristics of Indian coffee and tailoring your brewing to suit, you can enjoy a truly satisfying and personalized coffee experience. Explore and Experiment: Don’t hesitate to try different beans, roast levels, and brewing methods. Appreciate the Craft: Recognize the effort at each stage of the coffee’s journey. While this guide reflects my personal exploration and understanding of Indian coffee. I hope it inspires you to embark on your own coffee adventure!\nReferences Coffeeboard India: regions Coffeeboard India: statistics Coffee in India Wikipedia Multiple blogs from speciality roasters of India The coffee I had Kokoro Kensho SLN 13, SLN 9, SLN 795 The cafe in Pune Goddam speciality coffee ",
  "date": "Feb 1, 2025",
  "tags": [
   "life"
  ],
  "title": "Coffee Beans of India, Varieties and Taste Profiles",
  "url": "https://pankajpipada.com/posts/2025-02-01-indian-coffee-beans/"
 },
 {
  "content": "In this post, we will explore several excellent resources for understanding Retrieval-Augmented Generation (RAG). The post is structured as a step-by-step learning journey. This is intended to be a regularly updated list.\n101 - Basic introductions a) Microsoft’s Generative AI for Beginners RAG chapter b) AWS short article What is RAG c) NVidia’s introduction blog 201 - Depth over introductions a) NVidia’s Technical Brief b) NVidia’s Building RAG Agents with LLMs c) Google’s Cloud Skills Boost course d) LLamaIndex Understanding RAG 301 - In depth explorations Blogs\nSystematically Improving Your RAG blog LLMs and Graphs Part 1 blog LLMs and Graphs Part 2 blog OpenRAG paper Output Evaluation paper Data to SQL paper ReACT paper ReWoo paper Tree retrievals\nRaptor Tree retrieval paper RAGFlow github repo Knowledge graphs\nKG Retriever paper GraphRAG Survey paper Microsoft GraphRag\nIntro - blog Github repo Fast GraphRAG github repo LightRAG github repo NanoGraph github repo GraphDB backends\nNeo4j NaLLM github repo HippoRAG (graph + pagerank)\nHippoRAG github repo HippoRAG paper Interleaving Retrieval with Chain-of-Thought Reasoning\nIRCoT github repo IRCoT paper Memory based RAG\nPaper at arxiv MemoRAG github repo File document processing\nWDoc github repo Paperless-NGX docs Docling github repo ReRank\nFlashRank github repo Code Search\nSourcegraph website Frameworks: UI/API\nRAG to Riches R2R github repo R2R Docs website Doc Chat with UI Kotaemon github repo Timescale PostGreSQL AI github repo Collections\nRetrieval LM Papers github repo Awesome RAG github repo Rag Techniques github repo Semantic search companies Glean Guru Quench Coveo Yext BloomFire Qatalog ",
  "date": "Dec 17, 2024",
  "tags": [
   "gpt"
  ],
  "title": "Great Resources for Learning and Implementing Retrieval-Augmented Generation (RAG)",
  "url": "https://pankajpipada.com/posts/2024-12-17-rag-resources/"
 },
 {
  "content": "Links Emotional Design article by Donald Norman Book Notes/Excerpts Attractive things/Aesthetically pleasing -\u003e Positive emotions -\u003e Ease of use / Better function\n…We have long known that when people are anxious they tend to narrow their thought processes, concentrating upon aspects directly relevant to a problem. This is a useful strategy in escaping from danger, but not in thinking of imaginative new approaches to a problem. …tendency to repeat the same operation over again is especially likely for those who are anxious or tense. Levels of processing: visceral (automatic/wired), behavioral and reflective (contemplate)\n…it (reflective level) does not have direct access either to sensory input or to the control of behavior. Instead it watches over, reflects upon, and tries to bias the behavioral level. Design, pleasure and anxiety\n…Designers can get away with more if the product is fun and enjoyable. …Design – and for that matter, most problem solving – requires creative thinking followed by a considerable period of concentrated, focused effort. In the first case, creativity, it is good for the designer to be relaxed, in a good mood. Thus, in brainstorming sessions, it is common to warm up by telling jokes and playing games. No criticism is allowed because it would raise the level of anxiety among the participants. Good brainstorming and unusual, creative thinking require the relaxed state induced by positive affect. …Once the creative stage is completed, the ideas that have been generated have to be transformed into real products. Now the design team must exert considerable attention to detail. Here, focus is essential. One way to do this is through deadlines just slightly shorter than feel comfortable. Here is the time for the concentrated focus that negative affect produces. This is one reason people often impose artificial deadlines on themselves, and then announce those deadlines to others so as to make them real. Their anxiety helps them get the work done. … too much anxiety produces a phenomenon known as “tunnel vision”: the people become so focused that may fail to see otherwise obvious alternatives. Prewired, ready for adaptation and experiences\n…some biological mechanisms are only predispositions rather than full-fledged systems. …Children do not come into the world with language, but they do come predisposed and ready. ",
  "date": "Dec 17, 2024",
  "tags": [
   "papers"
  ],
  "title": "Emotional Design Article by Donald Norman",
  "url": "https://pankajpipada.com/posts/2024-12-17-emotion-and-design/"
 },
 {
  "content": "Task Management: A Constant Evolution Effective task management is crucial in software development, particularly for solo developers who manage all aspects of a project. Over the years, I’ve experimented with various systems—Zettelkasten, Eisenhower Matrix, Agile methodologies, and more—each offering unique benefits. In my previous blog post , I detailed a structured prioritization system that balanced immediate tasks with longer-term goals. While this approach was effective for managing multiple responsibilities, it became clear that a more streamlined system was necessary as my focus shifted toward solo software projects.\nIn solo projects, where you serve as the developer, project manager, and sometimes even the client, task management needs to be as simple and fluid as possible. The goal is to maintain focus on coding and problem-solving rather than managing too many tools. This realization led me to refine my system, integrating task management directly into my coding workflow using a simple, markdown-based approach.\nThe Workflow To streamline my workflow, I categorize tasks into four main priorities (as described in my previous post), a system that has proven effective across different contexts:\nP-0 (Immediate Tasks): The most urgent tasks that needed to be tackled right away. This list is kept very short to avoid overwhelming stress and serves as a critical indicator of workload and pressure. P-1 (Sooner than Later): Important but not immediate tasks that are next in line after P-0. These tasks typically feed into P-0 during calmer periods. P-2 (Obligations): A catch-all for tasks with no pressing urgency. These are often obligatory tasks that lack clear motivation, leading them to become “write and forget” items. P-X (Excitements): A short list of tasks that genuinely excite me, crucial for maintaining motivation. Tasks here had to be interesting enough to merit their inclusion. Each day, I draw from these categories to form (at least in my head) two lists:\nDo: What I plan to accomplish that day. Look forward to: Tasks that spark enthusiasm. This daily focus ensures that I stay on top of urgent work while also engaging with tasks that keep me motivated. The workflow works well for managing multiple streams of work, balancing motivation, and handling obligations.\nThe System Challenges with previous MS OneNote based system\nAs I transitioned to solo development, the need for a more integrated and fluid execution system became apparent. Managing tasks through MS OneNote caused frequent context switches, often disrupting my flow state. Also, the subpar support for Linux was a bit problematic. Switch to Markdown+Git:\nI had experimented with markdown-based task management before, but it wasn’t until I faced these challenges that I fully embraced it. Markdown, combined with Git, became the system that brought everything together. This approach integrates task management directly into the coding environment, minimizing context switches and keeping me in a productive flow state. With markdown, I manage different types of notes and tasks in a way that aligns with my workflow but is more streamlined. Ideas/Brainstorming/High-Level Deliberations For non-project-specific tasks or high-level ideas, I follow this implementation:\nInbox: Everything starts in an inbox.md file, which acts as a simple task list. Items here are sourced from random notes in Google Keep, spontaneous thoughts, or ad-hoc notes taken with pen and paper. Todo: Items move from inbox.md to a /todo.md which is a prioritized task list according to the previous workflow P0/PX/P1/P2. Ideas or refinements of thoughts or brainstorming’s dont move here. Ideas: These items are moved to specific files like ideaOrNoteTypeTitle/someFileName.md or /notes/ideaOrNoteTypeTitle.md. These files are generally similar to P-X lists, focusing on ideas that excite me but aren’t immediately actionable. These files are created as needed, and generic catch-all files (e.g., notes.md) are avoided. Sample folder structure: 1root/ 2├── inbox.md 3├── todo.md 4├── notes/ 5| ├── new-thought-1.md 6| └── meeting-notes.md 7└── my-grand-idea-1/ 8 ├── what-i-want.md 9 └── components.md I experimented with Foam Bubble , a tool that enhances markdown files with features like back-linking and graph visualization. However, in practice, I found that a well-organized files and folders graph was sufficient for my needs. Non-actionable items or notes kept for the sake of it tend to clutter my mind. Over time, I’ve realized that I’m truly interested in only a handful of topics. Fleeting thoughts often accumulate in my inbox.md (previously the P-X or P-2 box), but I rarely act on them. Discarding these after a period has helped me maintain a clear head-space.\nAlso, I have kept these files in my private GitHub that contains my blogs content. This has had its additional benefits, with one of the main advantages being keeping all my content in one place.\nProject-Specific Tasks For project-related tasks, I maintain a single /todo.md file within each project’s repository. This approach eliminates the need for context switching, allowing me to manage tasks directly within the development environment. Markdown’s simplicity—just plain text with minimal formatting—makes it an ideal tool for this purpose.\nThe new system maintains the core principles of my workflow but is more integrated and efficient for project management.\nProject Overview:\nEach project begins with a high-level overview in the todo.md file. This section serves as a brief, bullet-pointed design document, outlining the project’s goals and scope, particularly focusing on what’s necessary to reach the MVP (Minimum Viable Product). This overview is the foundation, and categorized as P-0 and P-1 system, but it is now directly tied to the project’s progression. Aim is to first burn through these tasks first and at speed without a lot of distractions. During this run, the subsequent sections get populated. Again, until this is done, nothing from next things get worked on. Running Laundry List:\nAs development progresses, new tasks naturally arise. These tasks are added to a “running laundry list” within the markdown file. This list is dynamic, constantly evolving as the project moves forward. It is categorized as P-0/P-1 only and anything P-2 is added to the next section. I tend to mix/re-prioritize things that are P-1 as P-0 if they excite me to some extent (i.e my original P-X). Pushed Out (i.e Backlog / Future Unknown):\nTasks that are still relevant but not immediately actionable are moved to a “Pushed Out” section—similar to a backlog but with a more critical filter. This section corresponds to P-2, focusing on future possibilities that excite me or hold significant potential. I tend to visit this section at some larger interval. Like my inbox, I have preferred to completely discard things from here if they have accumulated for quite some time. This avoids overthinking and enables a clean head-space. Challenges and Adjustments Mobile support:\nOne significant issue is mobile support. Editing markdown files on mobile devices, especially within Git repositories, can be awkward and frustrating. I experimented with GitJournal and other similar tools designed to manage Git-based markdown files on mobile, but the experience was not great. Editing in a mobile environment often felt jittery and unreliable, which disrupted the otherwise smooth workflow. Also, editing markdown text on mobile and running the git workflow with it, just doesn’t feel right. For quick, on-the-go notes, I use Google Keep, which I later transfer to markdown files. These notes are later transferred to the appropriate markdown files on my desktop, maintaining a single source of truth for all my tasks and ideas. For reading my current notes, GitHub mobile app is more than sufficient. Maybe there will be some bridge that will block a lot of git updates etc from mobile but still allow a smooth inbox population. While a more refined mobile solution would be helpful, this current setup is sufficient and doesn’t pose a significant barrier to my productivity. Daily list\nI have moved away from creating an actual list for daily things (i.e Do and Look forward to). It just sits as two bullets in my todo.md files so that I remind myself of the need to do it that way, but doing daily lists just doesn’t work out well. The items are still present, but “virtually” in my head. Tool choice\nLot of folks seem to suggest that Obsidian will give me best of all worlds, but that has not been the case for me as of now. May be I need to take a leap of faith at some point I suppose. Personal life\nLike last time, none of these systems have worked for my personal life. For personal tasks, I find an ad-hoc approach works best, with any minimal needs managed in Google Keep. Final Thoughts Markdown’s minimalism is its strength. Without the distraction of unnecessary features, I can organize tasks in a way that aligns perfectly with my workflow. It is just another text file with some styling indicators. The categories of P-0, P-1, P-2, and P-X are still present but simplified and their movement is more fluid.\nAs my work continues to evolve, this system will likely adapt with it, but for now, it strikes the right balance between structure and simplicity, keeping me productive and in the flow. It somehow, “feels right”.\n",
  "date": "Aug 13, 2024",
  "tags": [
   "life"
  ],
  "title": "Refining the Flow: A Streamlined Markdown/Git-Based Task Management System for Solo Developers",
  "url": "https://pankajpipada.com/posts/2024-08-13-taskmgmt-2/"
 },
 {
  "content": "Recently, I have moved to a home office as my primary workspace. Post COVID forced remote work, this is the first time I am really working from a home base. While a home office setup has a lot of benefits (no commute !!!), it introduces its own challenges. The major one out of it is the increase the sedentary lifestyle due to limited movement. Increase in back niggles, a general increase in procrastination, and lessened focus was the lesson learned during COVID times for me personally.\nSit-stand desks have become popular since COVID. There is an associated promise of reducing lethargy and increased focus with them. I finally decided to invest in it. I didn’t have a specific brand in mind but knew I wanted something reliable and easy to use. After a lot of research, I chose the IKEA Mittzon . It ticked all the boxes for motor reliability, stability, minimal vibrations when standing, and a smooth height adjustment and memory mechanism.\nThe assembly was surprisingly straightforward, turning a usually daunting task into an enjoyable evening project. Even as someone who isn’t a DIY enthusiast, I managed to put it together without any issues.\nIntegrating the sit-stand desk into my daily routine was a bit of a rollercoaster ride. At first, I switched between sitting and standing frequently. I realized that easing into standing, instead of making abrupt changes, helped in adjusting to the setup. Initially, my legs felt fatigued, and I hesitated to switch positions once I was comfortable. I also generally didn’t use house slippers/shoes generally when I sat all day, but that turned out to be a must for increasing my standing time. I spent time adjusting my monitor’s height and tilt, ensuring my arms and head were ergonomically aligned in both positions.\nAfter about a week, I found my groove. The desk’s memory function was incredibly helpful in quickly setting my preferred heights. I gradually increased my standing time, often standing while brainstorming or planning. For intense coding sessions, sitting was still my go-to. I thought standing might help curb procrastination by making me a bit uncomfortable, but that didn’t quite work out as planned.\nA month in, alternating between sitting and standing is not that hard. I definitely find my focus sharper during peak work hours. Being able to adjust my working position has also led to more overall movement throughout the day. I typically stand for about 30 minutes and sit for around an hour. My goal is to balance both for optimal comfort and productivity.\nThe learning curve of using a sit-stand desk was real, but the benefits for my physical well-being and productivity are undeniable. Standing while working has given me a new sense of mental clarity. The more I practice, the better it gets. It’s all about finding what works best for your body and workflow.\n",
  "date": "Jun 3, 2024",
  "tags": [
   "life"
  ],
  "title": "Personal experience with adapting to a Sit Stand Desk",
  "url": "https://pankajpipada.com/posts/2024-06-03-sit-stand-desk/"
 },
 {
  "content": "Note that the below snippets are stylized to Bootstrap 5. These can be easily adopted to any styling need.\nBreadcrumbs Partial 1\u003c!-- layouts/partials/breadcrumbs.html --\u003e 2\u003cdiv id=\"breadcrumbs-container\" style=\"height: 1.2em; line-height: 1em; margin: 0; overflow: hidden; font-size: small;\"\u003e 3\u003cul id=\"breadcrumbs\" style=\"list-style: none; padding: 0;\"\u003e 4 {{ range $index, $element := .Ancestors.Reverse }} 5 {{if not $element.IsHome }} 6 \u003cli class=\"text-muted\" style=\"display: inline;\"\u003e 7 \u003cspan\u003e\u0026gt;\u003c/span\u003e 8 \u003ca class=\"text-muted\" href=\"{{ $element.RelPermalink }}\"\u003e{{$element.Title}}\u003c/a\u003e 9 \u003c/li\u003e 10 {{else}} 11 \u003cli class=\"text-muted\" style=\"display: inline;\"\u003e 12 \u003ca class=\"text-muted\" href=\"{{ \"/\" | relLangURL }}\" aria-label=\"home\"\u003e\u003ci class=\"fas fa-home\"\u003e\u003c/i\u003e\u003c/a\u003e 13 \u003c/li\u003e 14 {{ end }}{{ end }} 15 {{ if not .IsHome }} 16 \u003cli class=\"text-muted\" style=\"display: inline;\"\u003e 17 \u003cspan\u003e\u0026gt;\u003c/span\u003e 18 \u003c/li\u003e 19 {{ end }} 20\u003c/ul\u003e 21\u003c/div\u003e Use as 1\u003c!-- Uncomment below --\u003e 2{{/* partial \"breadcrumbs.html\" . */}} Table of contents Partial 1\u003c!-- layouts/partials/toc.html --\u003e 2\u003cdiv class=\"mt-1 mb-5\"\u003e 3 {{ if and (gt .WordCount 300 ) (ne .Params.toc false) }} 4 \u003caside\u003e 5 \u003ch4\u003eTable of contents\u003c/h4\u003e 6 {{ .TableOfContents }} 7 \u003c/aside\u003e 8 {{ end }} 9\u003c/div\u003e Use as 1\u003c!-- Uncomment below --\u003e 2{{/* - partial \"toc.html\" . - */}} Code file as markdown Shortcode 1\u003c!-- layouts/shortcodes/codefile.html --\u003e 2{{ $file := .Get \"file\" }} 3{{ $lang := .Get \"lang\" | default \"plaintext\" }} 4{{ $content := readFile $file }} 5 6{{- /* Insert the content as a Markdown code block */ -}} 7{{ printf \"```%s\\n%s\\n```\" $lang $content | markdownify }} Use as 1\u003c!-- Uncomment below --\u003e 2{{/* % codefile file=\"posts/shell/zshrc\" lang=\"shell\" % */}} Quote Shortcode 1\u003c!-- layouts/shortcodes/authorquote.html --\u003e 2\u003cdiv class=\"authorquote border-start border-3 border-secondary px-4 py-1\" style=\"margin: 2.5em 10px; position: relative;\"\u003e 3 \u003cdiv class=\"quote-wrapper d-flex align-items-start\"\u003e 4 \u003ci class=\"fas fa-quote-left text-muted pe-2\" style=\"font-size: 2em; flex-shrink: 0;\"\u003e\u003c/i\u003e 5 \u003cdiv class=\"quote-content flex-grow-1\" style=\"line-height: 1.5;\"\u003e{{ .Inner }}\u003c/div\u003e 6 \u003c/div\u003e 7 \u003cdiv class=\"author text-muted text-end mt-1\" style=\"font-style: italic; font-weight: bold;\"\u003e-- {{ .Get \"author\" | markdownify }}\u003c/div\u003e 8\u003c/div\u003e Use as 1\u003c!-- Uncomment below --\u003e 2{{/* % authorquote author=\"Ian MacLaren\" % */}} 3Be kind, for everyone is fighting a hard battle. 4{{/* % /authorquote % */}} ",
  "date": "May 28, 2024",
  "tags": [
   "hugo"
  ],
  "title": "Hugo - Snippets for Breadcrumbs, Table of contents, Import code file as markdown, Quote",
  "url": "https://pankajpipada.com/posts/2024-05-28-hugo-breadcrumb-toc-codefile-quote/"
 },
 {
  "content": "One way to enhance the usability of your documentation or blog is by adding a copy button to your code blocks, allowing users to easily copy code snippets to their clipboard. In this post, we will integrate a copy button for code blocks in a Hugo-based static site.\nThe implementation involves creating a JavaScript function to add the button and then integrating that into your page layouts. The example provided uses Bootstrap 5 but can be adapted as needed.\nStep 1: JavaScript: Adding the Copy Button Create a new JavaScript file, copybutton.js with below code. This can be placed in the static/js or assets/js directory of your Hugo project. The implementation introduces a header row with language and copy button. It will do this for all code blocks with the language-* class and with some parent as highlight. The business logic can be easily modified to suit any other class layout (e.g: all pre code blocks, or all code blocks etc) In this current form, it would work for both table based class generation in hugo or normal one. Please note that you would need to adjust the styling to your theme. 1function addCopyButtonToCodeBlocks() { 2 // Function to determine if the background color is light or dark 3 function isColorDark(color) { 4 const rgb = color.match(/\\d+/g); 5 const r = parseInt(rgb[0], 10); 6 const g = parseInt(rgb[1], 10); 7 const b = parseInt(rgb[2], 10); 8 // Calculate luminance 9 const luminance = (0.299 * r + 0.587 * g + 0.114 * b) / 255; 10 return luminance \u003c 0.5; 11 } 12 13 // Function to adjust color brightness significantly 14 function adjustColorBrightness(color, amount) { 15 const rgb = color.match(/\\d+/g); 16 const r = Math.min(255, Math.max(0, parseInt(rgb[0], 10) + amount)); 17 const g = Math.min(255, Math.max(0, parseInt(rgb[1], 10) + amount)); 18 const b = Math.min(255, Math.max(0, parseInt(rgb[2], 10) + amount)); 19 return `rgb(${r}, ${g}, ${b})`; 20 } 21 22 // Get all code blocks with a class of \"language-*\" 23 const codeBlocks = document.querySelectorAll( 24 'pre \u003e code[class^=\"language-\"]' 25 ); 26 const copyIcon = '\u003ci class=\"fas fa-copy\"\u003e\u003c/i\u003e copy code'; 27 const copiedIcon = '\u003ci class=\"fas fa-check\"\u003e\u003c/i\u003e copied!'; 28 29 // For each code block, add a copy button inside a header 30 codeBlocks.forEach((codeBlock) =\u003e { 31 // Get the background color of the code block 32 const computedStyle = window.getComputedStyle(codeBlock); 33 const backgroundColor = computedStyle.backgroundColor; 34 35 // Adjust the header color to be significantly lighter or darker than the background color 36 const headerColor = isColorDark(backgroundColor) 37 ? adjustColorBrightness(backgroundColor, 65) 38 : adjustColorBrightness(backgroundColor, -65); 39 const textColor = isColorDark(backgroundColor) ? \"#d1d1d1\" : \"#606060\"; 40 41 // Create the header div 42 const header = document.createElement(\"div\"); 43 header.style.backgroundColor = headerColor; 44 header.style.display = \"flex\"; 45 header.style.justifyContent = \"space-between\"; 46 header.style.alignItems = \"center\"; 47 header.style.paddingRight = \"0.5rem\"; 48 header.style.paddingLeft = \"0.5rem\"; 49 header.style.borderTopLeftRadius = \"5px\"; 50 header.style.borderTopRightRadius = \"5px\"; 51 header.style.color = textColor; 52 header.style.borderBottom = `1px solid ${headerColor}`; 53 header.classList.add(\"small\"); 54 55 // Create the copy button 56 const copyButton = document.createElement(\"button\"); 57 copyButton.classList.add(\"btn\", \"copy-code-button\"); 58 copyButton.style.background = \"none\"; 59 copyButton.style.border = \"none\"; 60 copyButton.style.color = textColor; 61 copyButton.style.fontSize = \"100%\"; // Override the font size 62 copyButton.style.cursor = \"pointer\"; 63 copyButton.innerHTML = copyIcon; 64 copyButton.style.marginLeft = \"auto\"; 65 66 // Add a click event listener to the copy button 67 copyButton.addEventListener(\"click\", () =\u003e { 68 // Copy the code inside the code block to the clipboard 69 const codeToCopy = codeBlock.innerText; 70 navigator.clipboard.writeText(codeToCopy); 71 72 // Update the copy button text to indicate that the code has been copied 73 copyButton.innerHTML = copiedIcon; 74 setTimeout(() =\u003e { 75 copyButton.innerHTML = copyIcon; 76 }, 1500); 77 }); 78 79 // Get the language from the class 80 const classList = Array.from(codeBlock.classList); 81 const languageClass = classList.find((cls) =\u003e cls.startsWith(\"language-\")); 82 const language = languageClass 83 ? languageClass.replace(\"language-\", \"\") 84 : \"\"; 85 86 // Create the language label 87 const languageLabel = document.createElement(\"span\"); 88 languageLabel.textContent = language ? language.toLowerCase() : \"\"; 89 languageLabel.style.marginRight = \"10px\"; 90 91 // Append the language label and copy button to the header 92 header.appendChild(languageLabel); 93 header.appendChild(copyButton); 94 95 // Find the parent element with the \"highlight\" class and insert the header before it 96 const highlightParent = codeBlock.closest(\".highlight\"); 97 if (highlightParent) { 98 highlightParent.parentNode.insertBefore(header, highlightParent); 99 } 100 }); 101} 102 103// Call the function to add copy buttons to code blocks 104document.addEventListener(\"DOMContentLoaded\", addCopyButtonToCodeBlocks); Step 2: Integration in layouts To include this JavaScript file in your Hugo site, you need to modify a layout page where you want to have the copybutton present. If you want it to be present in all your pages, it could be the baseof.html file. If you place it in the assets/js directory, it can integrated as: 1{{ with resources.Get \"js/copybutton.js\" }} 2 {{ $minifiedScript := . | minify | fingerprint }} 3 \u003cscript src=\"{{ $minifiedScript.Permalink }}\" integrity=\"{{ $minifiedScript.Data.Integrity }}\" defer\u003e\u003c/script\u003e 4{{ else }} 5 {{ errorf \"copybutton.js not found in assets/js/\" }} 6{{ end }} If you place it in the static/js directory, it can integrated as: 1\u003cscript src=\"{{ \"js/copybutton.js\" | relURL }}\" defer\u003e\u003c/script\u003e ",
  "date": "May 28, 2024",
  "tags": [
   "hugo"
  ],
  "title": "Hugo - Introduce a Copy Button for Code Blocks",
  "url": "https://pankajpipada.com/posts/2024-05-28-hugo-copy-button/"
 },
 {
  "content": "In this guide, we will walk through the steps to integrate Datatables into a Hugo site. We will cover both node-based and CDN-based installations, and discuss the necessary page layout, table layout, and Datatable initialization functions.\nWhat Datatables Provide Datatables offer numerous features to enhance the user experience when working with tabular data:\nSorting: Allows users to sort data by clicking on column headers. Searching: Provides a search box to filter table data. Pagination: Splits the table into pages for easier navigation. Responsive Design: Ensures tables look good on different screen sizes. Customization: Extensive options to customize the appearance and behavior of tables. Step 1: Installation Datatables provides various styling and js options that supports jQuery, bootstrap and others. Below we talk about Bootstrap5 packages, but this can be adopted to any other styling system too.\nNode-Based Installation For a node-based Hugo setup, you can install Datatables and its dependencies using npm:\n1npm install jquery datatables.net datatables.net-bs5 After installation, ensure that the required JS and CSS files are included in your Hugo site’s assets.\nFirst, you would need to mount the node_modules to the assets folder. This can be done in config.toml:\n1[module] 2[[module.mounts]] 3source = \"node_modules\" 4target = \"assets/vendor\" Note that if you do this, you have to explicitly mount other folders too. Defaults are shown in the Hugo documentation here .\nNow you have to refer to the mount path (i.e., assets/vendor above) when using Hugo pipes to process this file. For example:\n1{{ $datatablesBS5CSS := resources.Get \"vendor/datatables.net-bs5/css/dataTables.bootstrap5.min.css\" }} 2\u003clink rel=\"stylesheet\" href=\"{{ $datatablesBS5CSS.RelPermalink }}\" /\u003e CDN-Based Installation For a simpler approach, you can use the CDN links to include Datatables and its dependencies:\n1\u003clink rel=\"stylesheet\" href=\"https://cdn.datatables.net/1.11.5/css/dataTables.bootstrap5.min.css\" /\u003e 2 3\u003c!-- These can be included in the specific page directly so that other pages are not impacted--\u003e 4\u003cscript src=\"https://code.jquery.com/jquery-3.6.0.min.js\"\u003e\u003c/script\u003e 5\u003cscript src=\"https://cdn.datatables.net/1.11.5/js/jquery.dataTables.min.js\"\u003e\u003c/script\u003e 6\u003cscript src=\"https://cdn.datatables.net/1.11.5/js/dataTables.bootstrap5.min.js\"\u003e\u003c/script\u003e Step 2: Content Page The content page defines the table structure to use with Datatables.\nFirst, you can create the content with a specialized layout. For example at: content/movies/_index.md.\nIf your actual data comes from a csv file, you can load that file and populate the table using a shortcode. Else, your data can directly go as content in the page.\nBelow is a sample content page and the associated csv read and table populate shortcode.\nSample content file:\n1--- 2title: 'Movie ratings' 3layout: 'movie' 4type: 'movie' 5toc: false 6summary: 'A table of my watched movies over the years and associated metadata along with my ratings.' 7--- 8 9\u003c!-- Uncomment this invocation of shortcode --\u003e 10 11{{/* \u003c csvtable src=\"movies/imdbratings_23072024.csv\" \u003e */}} --\u003e The csvtable shortcode defined at layouts/shortcodes/csvtable.html:\n1{{ $path := .Get \"src\" }} 2\u003c!-- Read the csv --\u003e 3{{ $csv := readFile $path }} 4{{ if $csv }} 5\u003c!-- Unmarshal it as a data object --\u003e 6{{ $data := $csv | transform.Unmarshal }} 7\u003c!-- Populate the table --\u003e 8\u003ctable id=\"movieRatingsDataTable\" class=\"display\"\u003e 9 \u003cthead\u003e 10 \u003c!-- Declare the columns for your table --\u003e 11 \u003ctr\u003e 12 \u003cth\u003eNo\u003c/th\u003e 13 \u003cth\u003eTitle\u003c/th\u003e 14 \u003cth\u003eMy Rating\u003c/th\u003e 15 \u003cth\u003eIMDb Rating\u003c/th\u003e 16 \u003cth\u003eDate Rated\u003c/th\u003e 17 \u003cth\u003eRelease Year\u003c/th\u003e 18 \u003cth\u003eDirectors\u003c/th\u003e 19 \u003cth\u003eGenres\u003c/th\u003e 20 \u003c/tr\u003e 21 \u003c/thead\u003e 22 \u003c!-- Populate rows --\u003e 23 \u003ctbody\u003e 24 {{ range $index, $row := $data }} 25 {{ if ne $index 0 }} 26 \u003ctr\u003e 27 \u003c!-- Note that the row object is dependent on your csv structure. Map proper column names. --\u003e 28 \u003ctd\u003e{{ $index }}\u003c/td\u003e 29 \u003ctd\u003e 30 \u003ca href=\"{{ index $row 5 }}\" target=\"_blank\"\u003e{{ index $row 3 }}\u003c/a\u003e 31 \u003c/td\u003e 32 \u003ctd\u003e{{ index $row 1 }}\u003c/td\u003e 33 \u003ctd\u003e{{ index $row 7 }}\u003c/td\u003e 34 \u003ctd\u003e{{ index $row 2 }}\u003c/td\u003e 35 \u003ctd\u003e{{ index $row 9 }}\u003c/td\u003e 36 \u003ctd\u003e{{ index $row 13 }}\u003c/td\u003e 37 \u003ctd\u003e{{ index $row 10 }}\u003c/td\u003e 38 \u003c/tr\u003e 39 {{ end }} 40 {{ end }} 41 \u003c/tbody\u003e 42\u003c/table\u003e 43{{ else }} 44\u003cp\u003eError: CSV file not found.\u003c/p\u003e 45{{ end }} Step 3: Page Layout The above page content now needs to be served using a specialized layout.\nThis layout includes the Datatables setup as well.\nThe name of the file and the layout name used in the content page should match.\nAdd the javascript dependencies here.\nFor example, layouts/_default/movie.html:\n1{{ define \"main\" }} 2\u003cdiv class=\"container\" style=\"max-width: 1080px; margin: auto; overflow-x: auto\"\u003e 3 \u003ch1 style=\"text-align: center; margin-bottom: 3rem\"\u003e{{ .Title }}\u003c/h1\u003e 4 \u003c!-- The .Content will be the page content loaded by Hugo --\u003e 5 \u003csection\u003e{{ .Content }}\u003c/section\u003e 6\u003c/div\u003e 7 8\u003c!-- Ensure scripts are deferred to improve page render performance --\u003e 9\u003cscript src=\"https://code.jquery.com/jquery-3.6.0.min.js\" defer\u003e\u003c/script\u003e 10\u003cscript src=\"https://cdn.datatables.net/1.11.5/js/jquery.dataTables.min.js\" defer\u003e\u003c/script\u003e 11\u003cscript src=\"https://cdn.datatables.net/1.11.5/js/dataTables.bootstrap5.min.js\" defer\u003e\u003c/script\u003e 12\u003cscript src=\"/js/movietables.js\" defer\u003e\u003c/script\u003e 13{{ end }} Step 4: Datatable Initialization Function The Datatable initialization is a javascript function.\nYou can write that as a separate script in static/js and use it, or add it in assets/js and load it via resource.Get as shown before or even embed it directly into the page layout file.\nAll your options related to page render can be added here.\nBelow is a simple example. This can be enhanced further as per your styling needs.\n1jQuery(function () { 2 $('#movieRatingsDataTable').DataTable({ 3 ordering: true, 4 order: [[4, 'desc']], // Sort by the \"Date Rated\" column, assumed to be the fifth column (index 4) 5 pageLength: 10, // Set the default number of rows per page 6 columnDefs: [ 7 { orderable: false, targets: [6, 7] }, // Make the Directors and Genres columns non-orderable 8 { className: 'dt-head-center', targets: '_all' }, // Center align all header cells 9 { className: 'dt-body-center', targets: '_all' }, // Center align all body cells 10 ], 11 pagingType: 'full_numbers', // Enhance pagination controls 12 responsive: true, // Enable responsiveness 13 scrollX: true, // Enable horizontal scrolling 14 language: { 15 search: 'Filter records:', // Customizing the search box text 16 lengthMenu: 'Display _MENU_ records per page', // Customize the length menu text 17 info: 'Showing page _PAGE_ of _PAGES_', // Customize the page information text 18 }, 19 }); 20}); Conclusion With these steps, you can effectively integrate Datatables into your Hugo site, providing a robust and user-friendly way to display tabular data. Example integration is present in this sites Movie ratings page here ",
  "date": "May 27, 2024",
  "tags": [
   "hugo"
  ],
  "title": "Hugo - Integrating Datatables",
  "url": "https://pankajpipada.com/posts/2024-05-27-hugo-datatables/"
 },
 {
  "content": "Adding search functionality to a static site generated with Hugo can significantly improve the user experience. lunr.js is a powerful JavaScript library for full-text search, offering a lightweight and fast solution ideal for static sites. In this guide, we will integrate search using lunr.js and address optimization concerns to ensure efficient and smooth operation. Note that the examples use Bootstrap and Font Awesome icons, but the same elements can be adapted to any styling system as required.\nStep 1: Create a data index for search In your config.toml, add the following lines to enable the generation of the search index file: 1[outputs] 2 home = [\"HTML\", \"RSS\", \"JSON\"] 3 4[outputFormats] 5 [outputFormats.JSON] 6 mediaType = \"application/json\" 7 baseName = \"index\" 8 isPlainText = true Then, create a layouts/index.json file, that will have a template for creating the data index. This file will be processed during hugo build to create a {output dir e.g public}/index.json For example, if you want to have title, url, content, tags, date available, the template will look something like below: 1{{- $index := slice -}} 2{{- range $.Site.RegularPages -}} 3 {{- $tags := slice }} 4 {{- range .Params.tags -}} 5 {{- $tags = $tags | append . }} 6 {{- end -}} 7 {{- $content := .Content | plainify | htmlUnescape }} 8 {{- $datestr := .Date.Format \"Jan 2, 2006\" }} 9 {{- $indexItem := dict \"url\" .Permalink \"title\" .Title \"content\" $content \"tags\" $tags \"date\" $datestr -}} 10 {{- $index = $index | append $indexItem }} 11{{- end -}} 12{{- $index | jsonify (dict \"indent\" \" \") }} Step 2: Create the Search Form Now that we have the data created for search, we would need to establish a interaction mechanism with the user. You can embed the search form in the header or body of all pages or restrict it to a dedicated search page. It can be embedded as {{ partial \"search-form.html\" . }} This form should, take input from user, and on action, invoke the search page with search query embedded into the url. Below is a slightly opinionated layouts/partials/search-form.html, using Bootstrap classes and Font Awesome icons for styling. This can be adapted to any styling system. The form takes user input and invokes the search page with the search query embedded into the URL. 1\u003c!-- Form with a get page action --\u003e 2\u003cform id=\"search\" action='{{ with .GetPage \"/search\" }}{{.Permalink}}{{end}}' method=\"get\" class=\"d-flex justify-content-center mt-2 mb-4\"\u003e 3 \u003c!-- Hidden label for accessibility --\u003e 4 \u003clabel hidden for=\"search-input\"\u003eSearch site\u003c/label\u003e 5 \u003cdiv class=\"input-group\" style=\"width: 90%;\"\u003e 6 \u003c!-- Icon inside the input group for visual enhancement --\u003e 7 \u003cspan class=\"input-group-text border-0 bg-transparent\"\u003e 8 \u003ci class=\"fa fa-search\"\u003e\u003c/i\u003e 9 \u003c/span\u003e 10 \u003c!-- Search input field. The \"name\" defined here will be used to parse the URL when executing the business logic in search.js --\u003e 11 \u003cinput type=\"text\" class=\"form-control rounded-pill\" id=\"search-input\" name=\"query\" placeholder=\"Type here to search...\" aria-label=\"Search\"\u003e 12 \u003c!-- Submit button with an arrow icon --\u003e 13 \u003cbutton class=\"btn border-0 bg-transparent\" type=\"submit\" aria-label=\"search\"\u003e 14 \u003ci class=\"fa fa-arrow-right\"\u003e\u003c/i\u003e 15 \u003c/button\u003e 16 \u003c/div\u003e 17\u003c/form\u003e Step 3: Set Up Your Search Content Page To actually execute the query and display the results we need to create a search content page. Generally it would be content/search/_index.md, but this can change depending on your chosen site organization for Hugo. This file should point to a search layout, that we will create below. You can customize it as required. Typically a bare minimum file will look like: 1--- 2title: \"Search\" 3layout: \"search\" 4description: \"Search page\" 5--- Step 4: Create the Search Layout Now, the above page needs to be served using a layout. The search layout defines the structure of the search page and includes necessary scripts for lunr.js and the custom search logic. By including these scripts in the layout page only, we can ensure that the search functionality is loaded on this page only and doesn’t really affect other pages. This can help optimize performance for the rest of the site. Create a search.html file in your layouts/_default directory: 1{{ define \"main\" }} 2\u003cdiv id=\"search-container\" class=\"container\"\u003e 3 \u003c!-- This is where the search results will be displayed. Initialize as empty list. --\u003e 4 \u003cul id=\"searchresults\"\u003e\u003c/ul\u003e 5\u003c/div\u003e 6 7\u003c!-- Include lunr.js library. This can be included from node_modules mounted as assets/vendor, refer to their cdn or directly use from static/js. --\u003e 8{{ $lunrJS := resources.Get \"vendor/lunr/lunr.min.js\" }} 9\u003cscript src=\"{{ $lunrJS.RelPermalink }}\" defer\u003e\u003c/script\u003e 10 11\u003c!-- Include the custom search script where the magic happens. This can be used from assets/js like below, or directly from static/js. --\u003e 12{{ with resources.Get \"js/search.js\" }} 13 {{ $minifiedScript := . | minify | fingerprint }} 14 \u003cscript src=\"{{ $minifiedScript.Permalink }}\" integrity=\"{{ $minifiedScript.Data.Integrity }}\" defer\u003e\u003c/script\u003e 15{{ else }} 16 {{ errorf \"search.js not found in assets/js/\" }} 17{{ end }} 18{{ end }} Step 5: Create the Search business logic script Now we need to connect all the above site elements to lunr.js, perform search and render results. We will create a javascript script for this. This script handles the entire search process, including loading the search index, processing search queries, and displaying results. The below script can be placed as assets/js/search.js and included in your search layout as shown in a previous step. Alternately, you can put it directly inside static/js folder too and include it via the search layout above. Flow of the Code Initialization: The script initializes the lunr.js search index and ensures it only happens once for a page load. Caching: It checks for cached search data in localStorage. If valid cached data is available, it uses it; otherwise, it fetches new data and caches it. Building the Index: The script constructs the lunr.js search index from the fetched data. Search Query Handling: It reads the search query from the URL parameters and triggers a search if a query is present. Search Execution: It performs the search using the built index and processes the query to ensure it is valid. Displaying Results: It limits the displayed results to a maximum of 10 to avoid overwhelming users and improve performance. 1// Get the search input element 2var searchElem = document.getElementById(\"search-input\"); 3// Define a global object to store search-related data and ensure it's initialized only once 4window.pankajpipadaCom = window.pankajpipadaCom || {}; 5 6// Initialize search only once 7if (!window.pankajpipadaCom.initialized) { 8 window.pankajpipadaCom.lunrIndex = null; 9 window.pankajpipadaCom.posts = null; 10 window.pankajpipadaCom.initialized = true; 11 12 // Load search data and initialize lunr.js 13 loadSearch(); 14} 15 16// Function to load search data and initialize lunr.js 17function loadSearch() { 18 var now = new Date().getTime(); 19 // Check for cached data in localStorage 20 var storedData = localStorage.getItem(\"postData\"); 21 22 // Use cached data if available and not expired 23 if (storedData) { 24 storedData = JSON.parse(storedData); 25 if (now \u003c storedData.expiry) { 26 console.log(\"Using cached data\"); 27 buildIndex(storedData.data, checkURLAndSearch); 28 return; 29 } else { 30 console.log(\"Cached data expired\"); 31 localStorage.removeItem(\"postData\"); 32 } 33 } 34 35 // Fetch search data via AJAX request 36 var xhr = new XMLHttpRequest(); 37 xhr.onreadystatechange = function () { 38 if (xhr.readyState === 4 \u0026\u0026 xhr.status === 200) { 39 try { 40 var data = JSON.parse(xhr.responseText); 41 buildIndex(data, checkURLAndSearch); 42 console.log(\"Search initialized\"); 43 44 // Cache fetched data with expiry 45 localStorage.setItem( 46 \"postData\", 47 JSON.stringify({ 48 data: data, 49 expiry: new Date().getTime() + 7 * 24 * 60 * 60 * 1000, // TTL for 1 week 50 }) 51 ); 52 } catch (error) { 53 console.error(\"Error parsing JSON:\", error); 54 showError(\"Failed to load search data.\"); 55 } 56 } else if (xhr.status !== 200) { 57 console.error(\"Failed to load data:\", xhr.status, xhr.statusText); 58 showError(\"Failed to load search data.\"); 59 } 60 }; 61 xhr.onerror = function () { 62 console.error(\"Network error occurred.\"); 63 showError(\"Failed to load search data due to network error.\"); 64 }; 65 xhr.open(\"GET\", \"../index.json\"); 66 xhr.send(); 67} 68 69// Function to build lunr.js index 70function buildIndex(data, callback) { 71 window.pankajpipadaCom.posts = data; 72 window.pankajpipadaCom.lunrIndex = lunr(function () { 73 this.ref(\"url\"); 74 this.field(\"content\", { boost: 10 }); 75 this.field(\"title\", { boost: 20 }); 76 // Define the new field for concatenated tags 77 this.field(\"tags_str\", { boost: 15 }); 78 this.field(\"date\"); 79 window.pankajpipadaCom.posts.forEach(function (doc) { 80 // Create a new field 'tags_str' for indexing 81 const docForIndexing = { 82 ...doc, 83 tags_str: doc.tags.join(\" \"), 84 }; 85 this.add(docForIndexing); 86 }, this); 87 }); 88 console.log(\"Index built at\", new Date().toISOString()); 89 callback(); 90} 91 92// Function to display error message 93function showError(message) { 94 var searchResults = document.getElementById(\"searchresults\"); 95 searchResults.innerHTML = `\u003cbr\u003e\u003ch2 style=\"text-align:center\"\u003e${message}\u003c/h2\u003e`; 96 searchElem.disabled = true; // Disable search input on error 97} 98 99// Function to check URL for search query and perform search 100function checkURLAndSearch() { 101 var urlParams = new URLSearchParams(window.location.search); 102 var query = urlParams.get(\"query\"); 103 if (query) { 104 searchElem.value = query; 105 showSearchResults(); 106 } 107} 108 109// Function to perform search and display results 110function showSearchResults() { 111 if (!window.pankajpipadaCom.lunrIndex) { 112 console.log(\"Index not available.\"); 113 return; // Exit function if index not loaded 114 } 115 var query = searchElem.value || \"\"; 116 var searchString = query.trim().replace(/[^\\w\\s]/gi, \"\"); 117 if (!searchString) { 118 displayResults([]); 119 return; // Exit if the search string is empty or only whitespace 120 } 121 122 var matches = window.pankajpipadaCom.lunrIndex.search(searchString); 123 console.log(\"matches\", matches); 124 var matchPosts = matches.map((m) =\u003e 125 window.pankajpipadaCom.posts.find((p) =\u003e p.url === m.ref) 126 ); 127 console.log(\"Match posts\", matchPosts); 128 displayResults(matchPosts); 129} 130 131// Function to display search results 132function displayResults(results) { 133 const searchResults = document.getElementById(\"searchresults\"); 134 const maxResults = 10; // Limit to 10 results 135 if (results.length) { 136 let resultList = \"\"; 137 results.slice(0, maxResults).forEach((result) =\u003e { 138 if (result) { 139 resultList += getResultStr(result); 140 } 141 }); 142 searchResults.innerHTML = resultList; 143 } else { 144 searchResults.innerHTML = \"No results found.\"; 145 } 146} 147 148// Function to format search result items 149function getResultStr(result) { 150 var resultList = ` 151 \u003cli style=\"margin-bottom: 1rem\"\u003e 152 \u003ca href=\"${result.url}\"\u003e${result.title}\u003c/a\u003e\u003cbr /\u003e 153 \u003cp\u003e${result.content.substring(0, 150)}...\u003c/p\u003e 154 \u003cdiv class=\"text-muted\" style=\"display: flex; justify-content: space-between; align-items: center; font-size: small; height: 1.2em; line-height: 1em; padding: 0.25em;\"\u003e 155 \u003cdiv\u003e${result.date}\u003c/div\u003e 156 \u003cdiv\u003e\u003ci class=\"fa fa-tags\"\u003e\u003c/i\u003e 157 ${result.tags 158 .map( 159 (tag) =\u003e 160 `\u003ca class=\"text-muted\" href=\"/tags/${tag}\"\u003e${tag}\u003c/a\u003e` 161 ) 162 .join(\", \")} 163 \u003c/div\u003e 164 \u003c/div\u003e 165 \u003c/li\u003e`; 166 return resultList; 167} Optimization Concerns Index Data Caching By default, the index is not preserved across page loads, which can result in unnecessary data fetching and processing. To improve performance, we can use localStorage to cache the search data. This caching mechanism is already implemented in the loadSearch function, where data is stored with a time-to-live (TTL) of one week. This ensures that the index is only fetched and built once a week, reducing the load on the server and improving user experience. Note that localStorage needs data to be in a serializable format and hence the index directly cannot be cached. Therefore, we are caching the index.json post data that we created in the first step and then rebuilding the index at each window creation. Limiting Search Results Limit the number of search results displayed to the user to avoid overwhelming them and to improve performance. This is done by taking a maxResults length slice above. Async Loading of Scripts To improve page load times, ensure that the search scripts are loaded asynchronously. This is achieved by adding the defer attribute to the script tags in the layout. 1\u003cscript src=\"{{ $lunrJS.RelPermalink }}\" defer\u003e\u003c/script\u003e 2\u003cscript src=\"{{ $minifiedScript.Permalink }}\" integrity=\"{{ $minifiedScript.Data.Integrity }}\" defer\u003e\u003c/script\u003e Note that this deferring means the page will first render and then the actual search execution will begin. Query-Based Search Our layout doesn’t really communicate with the script as such. It just loads the script. The script itself, sees if the data and indexes are present, then checks the URL for search query, then executes the search, modifies the html to add the result list items. It also helps to allow users to share searches easily. This is already handled in the checkURLAndSearch function, which reads the query parameter from the URL and performs a search if it’s present. Conclusion To recap, we created a search index data, a user interaction form, a search page and a layout for it. All this and lunr.js is tied together using a custom javascript. As noted before, the stylings used are bootstrap and font awesome based here, but can be easily adapted to any styling system. An implemented example of this can be found in this sites search functionality. Example search query ",
  "date": "May 27, 2024",
  "tags": [
   "hugo"
  ],
  "title": "Hugo - Integrate search using lunr.js",
  "url": "https://pankajpipada.com/posts/2024-05-27-hugo-search/"
 },
 {
  "content": "In Hugo, copying files to the output during the build process is a common task, especially when dealing with external resources like CSS, JavaScript, or source maps from node_modules. In this post, we will walk through how to achieve this without any additional external dependencies.\nResources Resources in Hugo are any files that can be processed by Hugo Pipes, such as images, stylesheets, and javascript files. These resources can be transformed, minified, concatenated, and otherwise manipulated using Hugo’s built-in functions. They are typically placed in the assets directory or other mounted directories. A common method to copy resources involves using resource.Get followed by Permalink, RelPermalink, or Publish. Here’s an example of how to copy a CSS file from node_modules to your output: First, mount node_modules to the assets folder. This can be done in config.toml:\n1[module] 2[[module.mounts]] 3source = \"node_modules\" 4target = \"assets/vendor\" Note that if you do this, you have to explicitly mount other folders too. Defaults are shown in the Hugo documentation here .\nNow you have to refer to the mount path (i.e., assets/vendor above) when using Hugo pipes to process this file. For example:\n1{{ $bootstrapCSS := resources.Get \"vendor/bootstrap/dist/css/bootstrap.min.css\" }} 2\u003clink rel=\"stylesheet\" href=\"{{ $bootstrapCSS.RelPermalink }}\"\u003e Non-Resources Non-resources are files that Hugo does not process automatically, such as certain source maps or configuration files.\nThese files might be crucial for development or debugging but are not part of the standard resource pipeline in Hugo.\nFor non-resources, the above approach doesn’t work directly because Hugo doesn’t create resources for them.\nInstead, you can follow these steps:\nRead the file. Use resources.FromString. Publish the resource. To optimize this process, you can create a partial and use it in your layout.\nHere’s an example partial that can be placed as layouts/partials/copy-sourcemap-from-nodemodules.html:\n1{{ $mapFileOrig := . }} 2{{ $mapFileNode := printf \"/node_modules/%s\" $mapFileOrig }} 3{{ $mapFileVendor := printf \"vendor/%s\" $mapFileOrig }} 4{{/* warnf \"Source map in: %s, node path: %s vendor path: %s\" $mapFileOrig $mapFileNode $mapFileVendor */}} 5{{ $mapContent := readFile $mapFileNode }} 6{{ if $mapContent }} 7 {{ $map := resources.FromString $mapFileVendor $mapContent }} 8 {{ $map.Publish }} 9{{ else }} 10 {{ errorf \"Source map not found: %s\" $mapFileNode }} 11{{ end }} To use this partial, include it in your template (generally baseof.html, etc) like this:\n1{{ partial \"copy-sourcemap-from-nodemodules.html\" \"bootstrap/dist/css/bootstrap.min.css.map\" }} Note that during development, you may need to deleted the output folder and then build your site again for resource copy to work.\nConclusion By following these steps, you can effectively manage and include both resources and non-resources in your Hugo projects, ensuring that all necessary files are available in the final output.\n",
  "date": "May 27, 2024",
  "tags": [
   "hugo"
  ],
  "title": "Hugo - Copying Files to Output Using Pipes",
  "url": "https://pankajpipada.com/posts/2024-05-27-hugo-copy/"
 },
 {
  "content": "In this post, we will explore several excellent resources for understanding LLMs. My focus is primarily text and code inference, but most of the things should be generic enough for everyone. The post is structured as a step-by-step learning journey. This is intended to be a regularly updated list.\n101 - Basic introductions a) Andrej Karpathy’s - Intro to Large Language Models This video provides a general and high-level introduction to LLMs, covering topics such as inference, scaling, fine-tuning, security concerns, and prompt injection. b) Nvidia’s Generative AI explained Note: This course would require you to login to nvidia first and then details are visible. This video gives a very high level overview of GenAI, its usage, applications that companies are targeting, etc. 102 - Courses that cover basic usage a) Microsoft’s Generative AI for Beginners I personally have found this to be a excellent course that touches LLMs, prompting, RAG, Agents, multi modals etc. b) Google’s Cloud Skills Boost Havent gone through this personally, but content looks ok. 103 - Prompt engineering Note that prompt engineering techniques may seem generic, but in practice every family of models generally have specific things that work for them. If you change models, the prompts may need to be adjusted for better results. (E.g Claude 2.x worked great with XML tags for constraints or 1 shot examples, GPT4 worked better with JSON)\na) Microsoft’s Introduction to prompt engineering b) OpenAI’s Prompt engineering guide c) Anthropic Claude’s Prompt engineering guide d) Dedicated Prompt Engineering Guide 201 - LLMs a) Andrej Karpathy’s State of GPT Covers things like tokenization to pre-training, supervised fine-tuning, and Reinforcement Learning from Human Feedback (RLHF). Also, practical techniques and mental models for the effective use of these models, including prompting strategies, fine-tuning, etc are covered. b) Visual Introduction: LLMs by Bycroft Excellent visualizations and explanations of LLMs using nanoGPT, GPT2, GPT3 Understanding the visualizations is slightly involved, most probably a 301 rather than 201, but glancing through also can help to some extent. 301 - Deep Dive into Gen AI / Machine Learning a) Andrew Ng’s Machine Learning Introduction The definitive course to dive into ML. Updated for covering GenAI too. (94 hrs) b) Fast.ai’s Course c) Andrej Karpathy’s Zero to Hero This is a youtube series that will build a GPT from scratch. d) 3Blue1Browns Season 3 - Neural networks Ideal course will be to take from season 1 to 4 to understand things end to end. e) Google’s Cloud Skills Boost for Advanced Devs I am skeptical about this learning path, but some folks have found this useful. 401 - Miscellaneous a) Stanford AI Courses b) Nvidias deep learning institute The institute has some interesting learning paths that are self paced. Free as well as paid content present. c) Tools and more extensive resources list at awesome-generative-ai Journey recommendation and conclusion It is important to tailor the learning journey based on personal interests and goals. If you are interested in consumption of LLM’s, having a basic understanding of concepts in 10x and 20x should be enough.\nFor folks interested specifically in text or code generation, I would recommend understanding below order of key areas:\nText Completion/Inference Prompt Engineering Tools/Function Calling RAG (Retrieve, Augment, Generate) Agents Fine Tuning Thinking of a use-case of interest and starting with coding it as early as possible along with above material gives best results in this authors experience.\nIf possible, avoid libraries (For Python, LLamaIndex and Langchain are the most popular ones as of now), when learning and try to write your own code. Calling APIs in python or any other language is generally very straight forward (chat gpt with gpt-3.5 should be able to give that code very easily).\nLibraries add a layer of abstraction that creates more impedance, than help during learning phase. I myself was able to pickup typescript, react, etc very easily using GPT’s and its APIs. One early experience is documented in this post . The open source FlexiGPT vscode plugin is an outcome of my personal AI journey.\nHappy learning !!!\n",
  "date": "Apr 15, 2024",
  "tags": [
   "gpt"
  ],
  "title": "Great Resources for Learning Generative AI and Large Language Models (LLMs)",
  "url": "https://pankajpipada.com/posts/2024-04-15-genai-resources/"
 },
 {
  "content": "Edit: A next stage evolution of this system can be found in my follow up post here Evolution In this ever-changing world, with shifting demands and responsibilities, I have experimented through multiple task management strategies. E.g: Zettelkasten (much more than task management, but still), Eisenhower matrix, Getting things done (GTD) method, Agile/Kanban, ABCDE method, Franklin planner method, etc.\nFinally, I’ve settled on a system that works for me (in most cases). It’s most likely a mash-up of various methods recommended from books, blog posts, conversations with experienced folks, etc. It may not be perfect and might even change in the future with changing roles and responsibilities, but it has served me well since quite some time.\nGranularity I mostly deal with unit items of tasks which are neither too small, nor too large.\n“Steps” to complete one item are not included (Not Too Small).\nProjects or ideas that might span a long time (months to years) are excluded (Not Too Large). These items and associated brainstorming don’t come into the task list. Parts of it only come here once these things materialize and are broken down into smaller units. Deliberation on a project, visions, designs, breakdowns into smaller items, are outside of task list.\nArrangement (prioritization) P-0: A very short list of things I would do “immediately”. Items that someone has told you to do immediately, but which you practically won’t be able to accomplish, should not be included (i.e. give me yesterday things). Stress Indicator: A large P-0 list means very high level of stress and is a sign of a problem that you need to address. P-1: Slightly larger list that contains the items that I need to address “sooner than later”. Many tasks enter directly into P-1. During peaceful times, P-0 is mostly populated from P-1. Although direct entries into P-0 can still happen, I prefer to avoid this as much as possible P-2: A large list of various obligations and tasks. These may be things I think I have to do, but they have no associated urgency and questionable motivation. This is an often neglected list. It turns out to be a “write and forget” list, with a hope attached that it “may” be used someday. General preference is to move things from here to associated project/ideas pages and reevaluate at that level rather than in task list. P-X: A short list of things that excite me. Keeping it short is crucial. If this grows, that means you I am not really excited by all of them and most probably few items need to be in P2. Daily focus (the heart of my system) Every day come up with tasks in two categories (~5/10 minutes daily time spend):\nDo: These are the tasks I have planned for the day, primarily a subset from P-0. Meetings are excluded from this for now. They tend to be obligatory and preplanned most of the time. I don’t see a lot of value in adding them as tasks. Look forward to: Items that excite me. Generally items come from P-0 or P-X here. If there are days when this list is empty, it’s a sign of a problem that I need to address. Good days are when there are more than 1 item here. Tools Tried and discarded:\nGoogle keep: It served well enough in very early stages. Somehow the “features” added increased and its usefulness decreased. I still use it for lists or things I need to refer to once in a while, but not for tasks. Roam research/Notion: Tried them for a very short time and most probably very early in their lifecycle. Never got over the initial barrier. Editor plugins Was interested in this as my writing in Markdown increased over time. Multi device availability and less suitability for projects/ideas deliberation were a problem. Kanban boards Tried it out just for the heck of it at some point. Looking at a board at the start of the day was a very bad feeling generally. Current:\nMicrosoft OneNote Got introduced to it via a colleague when I shifted to Mac from Linux. I can easily keep most things in one “Page”. Daily “Do/Look forward to” list can be created in minimal time. Moving around tasks is very easy. Project/ideas/Meeting notes pages can be kept in same place. Sync is available across my devices (Mac/Android). There are a lot of features there that I have almost never used. Hope is that the primary workflow and UX for note-taking remains more or less same. Parting note Interestingly, this model doesn’t apply to my family life at all. But, I find myself following it about 80% of the time in my work/career life. It’s a delicate balance between the immediate necessities and the sparks of excitement that keep me going.\nThe red flags, the problems, the overwhelming moments — all find their reflection in this system. It is not necessary that this model reduces stress per se. Depending on the way my career was going the P-0 list was very overwhelming at some points. And even though I was aware of the red flags, I couldn’t do much about it.\nI hope sharing this provides a window into my thoughts and methods, but it’s essential to remember that this is purely my experience. What works for me might not be applicable to anyone else, as it’s a product of my unique journey and professional life.\n",
  "date": "Jul 30, 2023",
  "tags": [
   "life"
  ],
  "title": "Finding my balance: An evolved and simplified task management system",
  "url": "https://pankajpipada.com/posts/2023-07-30-taskmgmt/"
 },
 {
  "content": "Blunt/Direct communicators A lot of engineers (especially individual contributors) get characterized as blunt or direct or assertive communicators. These individuals generally communicate their thoughts and opinions in an unfiltered and straightforward manner. The desired outcome from a conversation these folks expect is clarity. Sugar-coating, diplomatic phrasings, adding vagueness, etc. are generally seen as detrimental to the conversation. Getting straight to the point generally seems important to them, rather than dance around a topic.\nA lot of people on the receiving end of these communications start perceiving this as rudeness. Even though the intention to hurt or belittle others may not be present, a perception of sabotage or hurt ego is very common. The necessary directness added to the conversation generally also is looked up as a superiority complex or ego centric-ness.\nOnline communication and its challenges In the increasingly remote work environment world, video calls (zoom, teams or otherwise), instant messaging (slack, etc.) and emails e omnipresent and their adoption is increasing. Even though these modes of communication facilitate a lot of necessary buffers between individuals (very necessary for introverts, another quality that is very much present in the same sample space of individuals), it presents some unique challenges like:\nTone Misinterpretation: Emails and text-based communication platforms lack the intonation, pauses, and emphasis that vocal speech allows. This absence leads to misinterpretations, with straightforward comments often perceived as harsh or impolite. In my personal experience, this is the largest source of “escalations”. Perceived Lack of Empathy: Direct communication can be efficient, but it might also be viewed as lacking empathy or consideration for others feelings. Loss of Non-Verbal Cues: In face-to-face communication, a direct individual often relies on physical cues like body language or tone, to soften the impact of their words or gauge the listener’s reaction. These cues are generally absent in online communication. These leads to, often missing or misunderstood interpretation by the receiver about the intention behind the message. Delayed Feedback: The delayed responses in a lot of these communication modes, leads to misunderstandings or create uncertainty about how a message has been received. Overemphasis on Content Over Relationship: Blunt speakers often prioritize the message’s content over the relationship with the receiver. This can further aggravate feelings of disconnect or misalignment. Formality and Expectations: A blunt person’s tendency to be concise and to-the-point comes off as rudeness in an environment where a certain level of formality or ‘small talk’ or ‘diplomacy’ is expected. Suggested solutions The below solutions/strategies have been suggested to me multiple times and through multiple forums (performance reviews, 1:1 feedbacks, escalation RCAs, leadership trainings, GPT, etc)\nEmbrace Emotional Intelligence: Be aware of the emotional impact of one’s words and use this understanding to frame the communication in a more palatable way. My experience: This is hard, like really hard. It definitely works, but doing it itself is a large challenge. Leverage Emojis/GIFs: Emojis/GIFs can be used to inject humor, soften a statement, or show empathy, which can prevent misinterpretations. My experience: This helps a bit in instant messaging. It is detrimental in almost all other settings. Seek Feedback and Clarify: Checking in or seeking feedback with the receiver can clear up potential misunderstandings and help demonstrate the intention to communicate respectfully. My experience: Feasibility to do this itself is often very less. It mostly works with someone you have an already established rapport with. Utilize Video Calls: Video calls can be a substitute for fave-to-face communication. My experience: Not useful other than 1:1 conversations. Develop Your Written Communication Skills: The written word lacks the subtlety of spoken language. Thus, learning to use phrases that convey respect, empathy, and openness can help to soften the impact of a direct communication style. My experience: Definitely necessary to know how to communicate your point “clearly”. But asking to stop being direct in written communication is a circular solution i.e stop being direct. Things that worked Apart from the above experiences, the below things have worked for me in varied settings. (Still a long way to go…):\nDetailed write-ups Writing down proposals/discussion points in detail with proper context setting, options considered, conclusions, sending these docs in advance and then discussing these as needed helps a lot. However, the challenge is that few people want to read lengthy write-ups. Of course this is not applicable to all scenarios, but if done a bit tactically, gives great results. Audio calls with discussion being done on a written medium While discussing a topic, if there are written points, which are projected on the screen, it helps a lot in streamlining the discussion and getting the point across in a much more palatable way. If it is an open discussion, just projecting a blank doc/editor and a facilitator writing down points, actions, pros/cons etc. helps a lot in everyone staying on track and getting to a conclusion. Digital whiteboards/drawing tools have always given worse results with lots of frustrations for me. Avoid any direct personal remarks While not everyone who is direct does it, but it is very easy to call out a person for a mistake or lack of knowledge. This is never ever a good idea. Maybe some 1:1 setting, but mostly should be avoided. Avoid satire Again, may not be a general thing, but never works in an online communication setting. Parting thoughts Seeking efficiency, transparency and honesty can be valuable in many contexts, especially in situations that require decisiveness, clarity, and swift action. Though it has its own challenges, just remembering that there is a human on other end of it can help a long way. More often than not, both the parties want to achieve the same goal.\n",
  "date": "Jul 10, 2023",
  "tags": [
   "life"
  ],
  "title": "Blunt/Direct/Assertive communicators, online communication challenges and how to overcome them",
  "url": "https://pankajpipada.com/posts/2023-07-10-assertive/"
 },
 {
  "content": "Today, I delved into Paul Graham’s (PG) How to do great work essay. He provides great insights into the topic and extensively discusses it (and I mean extensively…). However, there seems to be an aspect that is not addressed in the conversation – the people who are in survival mode or have recently emerged from it and how this affects their ability to do great work.\nWhen someone is in survival mode, they are primarily focused on meeting basic needs such as food, shelter, and healthcare. This can make it challenging to pursue personal interests, hobbies, or ambitious career goals because so much energy and resources are dedicated to just getting by. It’s a significant issue that many people face. Those who have just come out of survival mode are confronted with the challenge of unlearning some of the habits that got them out of survival mode and try to adjust to the new state, often while still carrying the mentality of survival mode.\nLet’s take a look, one by one, at the article’s arguments and how the above two could affect how to do great work.\nThe four steps PG prescribes the following four steps:\nChoose a field that aligns with your natural aptitude and deep interest (leads to harder work and diligence). Learn about your chosen field until you reach the frontier of knowledge. (Hard work needed). Notice the gaps in knowledge that others might overlook. Explore these gaps (fractal buds), especially the ones that others aren’t interested in. (Hard work needed). Now if you are in survival mode, the first step needs to be modified to: “Choose a field that aligns with your natural aptitude, can help you get by, and you have a deep interest in”. If the deep interest part doesn’t directly correlate with the ‘sustain’ aspect, you will most likely prioritize sustainability; but try to cultivate your interest over time as you stabilize your life. It may not be that difficult if the aptitude part of it is preserved.\nFor those who have recently exited survival mode, they have likely made some compromises in the first step. The challenge lies in breaking free from the survival mentality and allowing themselves to pursue fields that may not immediately contribute to their survival but align with their interests and aptitudes.\nEmbracing curiosity and adaptability PG touches upon the intricate journey of discovering one’s true calling. He argues that understanding the essence of a profession requires immersion and experience, often leading to a long, overlapping process of exploration, learning, and self-realization. He prescribes engaging in a broad spectrum of experiences, fostering curiosity and openness to increase the chances of discovering one’s passion. PG argues that if a field fails to captivate you as you delve deeper, it’s likely not your true calling. The importance of adaptability and the courage to change paths when a more exciting opportunity presents itself is highlighted. Finally, he warns against the distractions of societal pressures and external influences.\nFor survival mode and the life thereafter folks, the most difficult part remains about avoiding distractions due to societal pressures. This gives rise to the additional constraint of needing to ‘get by,’ and a marked increase in the difficulty level of adaptability and courage. For the folks that are just out of survival, the fear of falling back to it is the biggest deterrent.\nOvercoming Inertia and Procrastination Subsequently, PG touches upon navigating the journey of work requiring strategic management of time and energy. He argues that overcoming the initial inertia to start work often requires self-deception, like underestimating project complexity. He states that completion of projects is vital as it often leads to the most valuable outcomes. Furthermore, he warns of per-project procrastination, which can disguise itself as productivity and suggests regular self-checks to help stay on track.\nThe strategy of self-deception can be particularly useful for those transitioning out of survival mode, who may face additional challenges in starting new projects due to the lingering survival mentality.\nExcellence, consistency and long term value PG highlights the importance of aiming for the best in your field. This can be a challenging but rewarding goal for those in survival mode and for those transitioning out of survival mode.\nConsistency in doing great work is crucial. It’s not about getting a lot done every day, but about getting something done consistently. This principle holds true even more for those in survival mode or transitioning out of it. Every small step taken consistently can lead to significant progress over time.\nPG encourages us to aim to create something that will still be valued in a hundred years. This long-term perspective can guide your work and help ensure its lasting impact. For those transitioning out of survival mode, this perspective can provide a beacon of hope and a powerful motivation to strive for greatness, helping them break free from the survival mentality.\nUnlearning misconceptions and embracing experience In the journey from survival mode to a state of thriving, PG emphasizes the need to shed misconceptions and embrace the wisdom of experience. He challenges the passive learning model ingrained by traditional education systems, advocating for an active approach where educators are seen as advisors rather than authority figures.\nFor those transitioning out of survival mode, this shift in perspective can be empowering. It encourages self-reliance and autonomy, essential traits when navigating life beyond survival. PG warns against seeking shortcuts or ‘hacking the test’ for success, a mindset often adopted in survival mode. Instead, he emphasizes that real achievement comes from addressing overlooked problems and producing quality work.\nPG also advises against depending on external validation or ‘gatekeepers’ for success. This is particularly relevant for those emerging from survival mode, who may be accustomed to seeking approval or assistance from others. Instead, he encourages focusing on self-improvement and producing quality work.\nFinally, PG highlights the importance of learning from both positive and negative examples and the value of transferring ideas from one field to another. This can be particularly beneficial for those transitioning out of survival mode, as it encourages flexibility and adaptability, key traits for thriving in new environments.\nThe influence of people In the pursuit of greatness, the surrounding people can significantly shape your journey. Colleagues who inspire and challenge you can stimulate your growth and push you towards your goals. As you transition from survival mode, it’s crucial to surround yourself with individuals who fuel your optimism and maintain high morale. This positive cycle can enhance your work and drive you towards success. In my small personal experience, individuals who you can look up to due to their own dedication, focus and optimism are a great source of positive energy. Moreover, individuals that are in similar financial position as you and still are driven by things other than money are great motivations.\nYour audience, even if small and dedicated, can provide the necessary motivation and feedback for continuous improvement. Their appreciation and support can be a powerful catalyst, especially when transitioning out of survival mode.\nWhile prestige can be appealing, it’s important to remember that the value of your work should not be solely determined by others’ opinions. Instead, focus on excelling in your chosen field and making it prestigious through your own efforts. Curiosity, a powerful guide, can lead you to new discoveries and achievements.\nIn this journey, the influence of the right people can be a game-changer.\nSummary As you transition from survival mode to a state of thriving to doing great work, remember that the discoveries are out there, waiting to be made. Embracing the right internal and external shifts can go a long way.\n",
  "date": "Jul 2, 2023",
  "tags": [
   "life"
  ],
  "title": "Survival mode, life immediately after and Paul Graham's How to do great work",
  "url": "https://pankajpipada.com/posts/2023-07-02-great-work/"
 },
 {
  "content": "Background I got into espresso a few years back. I started out simply enjoying an office machine espresso, but as my interest grew, I found myself venturing to local cafes (these were sparse in early days, but the number has grown over the years in my town) and eventually crafting my own brew at home, over a span of 7 to 8 years.\nAs of today, I own a Baratza Virtuoso Plus grinder And a Gaggia Classic Pro coffee machine. I generally keep on trying different types of coffee bean roasts and origins.\nAlthough I’ve not conclusively established a favorite, my preference leans toward medium roasts. My top choices for coffee beans include either Gianyar or Kintamani Bali and Antigua Guatemala. When it comes to darker roasts, Sumatra and Ethiopia are my go-to beans.\nBeing the lone espresso consumer in my household, I typically brew a couple of espresso shots in the morning and, occasionally, an additional pair in the afternoon. It’s essential to note that the brewing method may would differ based on the frequency of usage of the machine. The below recipe is for a single daily home brew.\nThe Espresso Recipe Start by measuring 14g of coffee beans for a double shot. Set the grind size on your Baratza Virtuoso Plus grinder to 6 for medium roasts. You may adjust it to 6 or 8 for dark roasts, depending on the specific bean. Proceed to grind the beans. Switch on your Gaggia Classic Pro machine. Attach an empty portafilter to the machine and let it warm up until the first “brew” light illuminates. Activate the brew button to dispense the first cup of hot water. Once the light is off, turn off the button. Allow the machine to heat up again for the second time, and repeat the process to brew out a second cup of water. Remove and dry the portafilter using a kitchen cloth. Add the freshly ground coffee to the portafilter, and use a coffee tamper to compact the grinds. For best results, use a heavy tamper that fits the portafilter size perfectly. Reattach the filled portafilter to the machine. Activate the steam button to preheat the machine for 15 seconds. Turn off the steam button. The brew light will immediately illuminate. Turn on the brew button to start brewing. Allow the brew to process for about 20 seconds. Ideally, your brew should be a dark brown color, topped with a rich crema. I hope you enjoy this espresso journey as much as I have. Whether you’re a seasoned coffee lover or a newcomer, the beauty of coffee making is in the experience and the flavorful sip you earn at the end. Happy brewing!\n",
  "date": "Jun 2, 2023",
  "tags": [
   "life"
  ],
  "title": "My Go-to Espresso Recipe - An Everyday Brew",
  "url": "https://pankajpipada.com/posts/2023-06-02-espresso/"
 },
 {
  "content": "Dear friend,\nI am not sure why I am writing this now. I have thought about it many times, but never really got around to doing it. You have been on my mind since a few days now for some reason, and I have finally decided to write something.\nIt’s been 2 years 7 months on this day. I am not sure if I have processed things cleanly until now. I would most probably never be able to forgive myself for not doing more to help you. I am very sorry for that. Most probably I have had a lot of excuses for it.\nYou were a great friend. You were always trying to help others find happiness and make them feel comfortable. Mostly because of it, you had so many friends to share your good times. We did not share the same interests/vices, but I always felt great comfort in knowing that we could talk to each other freely and mostly about any topic under the sun, personal or otherwise. But, as I learned about some things that you were dealing with, I am not so sure anymore. I wish I could have given you more confidence so that you would open up about those things too with me. I wish I would have taken that COVID pass and come to meet you. I wish you never left our company. I wish you never moved out from your brothers place.\nThank you for becoming a friend in a sea of colleagues. Thank you for teaching me your out of the world debugging skills. Thank you for helping me with communicating more openly with people.\nI pray that you have found your peace.\nYour friend.\n",
  "date": "May 25, 2023",
  "tags": [
   "life"
  ],
  "title": "Remembering my friend",
  "url": "https://pankajpipada.com/posts/2023-05-25-remembering/"
 },
 {
  "content": "Introduction Have you ever wanted to learn a new programming language or framework but felt overwhelmed and unsure of where to start? Pair programming can offer a solution to this challenge by allowing you to learn from a more experienced developer in real-time. In this case study, we will explore the pair programming experience between me and ChatGPT in the context of learning about “Admonitions” in Hugo.\nRelevant background I have some small amount of experience in web development, and I am familiar with using Hugo, a popular open-source static site generator. My primary day job experience is with backend technologies and architecture. I am an absolute beginner when it comes to CSS styles, placing and using them in hugo, writing HTML using styles and hugo specific functions.\nI have been wanting to add an admonition that adds each quote in a box, to my quote’s page in this blog for some time. With the very limited experience I have in this area, I was not able to create a clean solution for it until now. With rise in ChatGPT’s and some experimentation with it, I decided to take on this task.\nWhat are Admonitions in Hugo? Admonitions are blocks of text that emphasize particular information and enhance the visual appeal of a document. They are commonly used to create notes, warnings, tips, and other types of annotations. Admonitions in Hugo can be created using the “shortcodes” feature, which allows you to add custom content to a page using a simple syntax.\nInteraction with ChatGPT Step 1: Set context Me \u003e Explain admonitions in hugo, give samples related to it, and guide on how to create them. (This was through multiple questions, but no code from my side.)\nChatGPT \u003e ChatGPT provides step-by-step instructions on how to create Admonitions in Hugo using the built-in shortcodes feature and custom CSS styles. I can see an example of adding quotes with HTML and associated CSS, as well as modifying the CSS to highlight the author name differently. ChatGPT also provides HTML code for creating a blockquote shortcode and modifying it to take the author name as an argument rather than as a separate span.\nStep 2: Get code for my problem with input code hints Me \u003e Style the blockquote admonition to match my hugo theme. This also through multiple questions, but this time, relevant code was provided each time.\nChatGPT \u003e I could learn how to modify the CSS styles to match my specific Hugo-themed-Bootstrap theme. ChatGPT provides CSS code for styling the blockquote elements to match the styles used in the theme, including the background color, border color, padding, and font styles for the author name.\nMe \u003e I was unsure on exact placement of CSS files and using them in shortcode in the theme. I asked ChatGPT about it.\nChatGPT \u003e ChatGPT also explains where to add CSS styles to a shortcode in Hugo. I could also learn about the steps to create a CSS file and include it in the HTML template for the site. This time too it gives relevant code blocks.\nStep 3: Non-ChatGPT steps needed At this point, I had a working code, but the font awesome icons were missing in the output as the theme had the icon in fas set and ChatGPT’s response had been guiding me to use it from the fa set. Identifying and rectifying them through ChatGPT interaction turned out to be very difficult. I had to go through the general route of exploring themes code, Stack Overflow search through Google, and then fixing it.\nStep 4: Block edit using ChatGPT instead of regex Me \u003e At this point a fully functional and tested shortcode was created. I now needed to use it across all the different quotes I had. Without ChatGPT, I would need to do a regex replace for doing some edits and then do some manual edits where regex was not really possible due to the slightly ad hoc nature of placement of information. I asked ChatGPT to do what I wanted as plain text command and provided input of my full text file.\nChatGPT \u003e ChatGPT was able to do the task of replacing the old “highlight” built-in admonition, with blockquote admonition, replace the author name from previous text field to the admonition parameter field perfectly. It was not able to reformat a few quotes into Markdown again properly. I decided to manually do that as it was a very minor task.\nExperience on guiding ChatGPT You can see that I needed to guide ChatGPT to the correct answer by asking specific and clear questions related to the topic of Admonitions in Hugo.\nI started by asking the basic question, then basic code, then edits to the code, then rectifying the code to match the specific environment. Even though ChatGPT helped to a large extent, identifying issues and whether the code generated and guidance is relevant to the consumers’ environment is definitely the user’s responsibility. Supplementing with additional context can assist, but the responsibility and the need for user knowledge remains unchanged.\nConclusion This interaction highlights the benefits of pair programming and learning, as it allows for a back-and-forth exchange of information and knowledge. Pairing with ChatGPT can be an effective way to learn about greenfield or brownfield programming concepts for a user.\nThough ChatGPT provides clear and concise instructions, examples, and code snippets to help the user understand and implement the concepts, as a consumer you need to be aware of what you want and what environment you are working with to get better results. There are some areas, where traditional debug and search Google/Stack Overflow is much easier path to solve the problem. Knowing when to move to the traditional method rather than spending time in ChatGPT is a skill to develop.\nWhether you are a beginner or an experienced developer, pair programming with ChatGPT can be a valuable learning experience.\nEpilogue This post itself was reviewed and modified using ChatGPT for spelling, grammar, sentence formulation, and structure. This task went fully without any hassle.\n",
  "date": "Feb 11, 2023",
  "tags": [
   "gpt",
   "hugo"
  ],
  "title": "Beginner level learning and pair programming with ChatGPT - A case study of Admonitions in Hugo",
  "url": "https://pankajpipada.com/posts/2023-02-11-pairprogram/"
 },
 {
  "content": "Links Amazon DynamoDB at USENIX ATC 22 Amazon Dynamo at SOSP 2007 Notes Good insight into how the evolution worked from Dynamo to DynamoDB.\nHaving a fixed unwavering goal of providing a managed service with fast and predictable performance at any scale is great.\nSystem properties DynamoDB is a fully managed cloud service. DynamoDB employs a multi-tenant architecture DynamoDB achieves boundless scale for tables DynamoDB provides predictable performance. DynamoDB is highly available DynamoDB supports flexible use cases Lessons learnt from the evolution Adapting to customers’ traffic patterns to reshape the physical partitioning scheme of the database tables improves customer experience. Performing continuous verification of data-at-rest is a reliable way to protect against both hardware failures and software bugs in order to meet high durability goals. Maintaining high availability as a system evolves requires careful operational discipline and tooling. Mechanisms such as formal proofs of complex algorithms, game days (chaos and load tests), upgrade/downgrade tests, and deployment safety provides the freedom to safely adjust and experiment with the code without the fear of compromising correctness. Designing systems for predictability over absolute efficiency improves system stability. While components such as caches can improve performance, do not allow them to hide the work that would be performed in their absence, ensuring that the system is always provisioned to handle the unexpected. ",
  "date": "Oct 18, 2022",
  "tags": [
   "papers"
  ],
  "title": "DynamoDB paper",
  "url": "https://pankajpipada.com/posts/2022-10-18-dynamodb/"
 },
 {
  "content": "Recently a friend posed a question to our common group: What common problems do you face in building complex, evolving, maintainable systems? Below is the general path that this discussion flowed.\nBroad level architectural thought Main Architectural goal for building large, complex enough and evolving systems is almost always the same: Minimize the resources (people, machines) needed to accommodate change.\nTop level method for doing this is almost always: separation of concerns. Achieving separation of concerns needs you to make tradeoffs. These are people, process, product related. E.g: dev velocity, team coordination, system performance, scalability, availability, failure models, etc.\nAs your org/project grows, the optimal tradeoff point shifts and you do the corresponding changes to adjust to these shifting tradeoffs.\nMulti service integration thought The defined API contract needs to be designed so that it is stable. Proper Resource based rest APIs come in handy for this. This is generally a non-trivial, error prone task for a lot of people, as defining resources you are handling and operations on them for today and tomorrow is very difficult. Same goes for DB schema design in a single service context.\nOne school of thought says that don’t worry about tomorrows responsibility as it is impossible to predict. While a good advice, completely ignoring any forward compatibility thought leads to a lot of pain down the line is a general observation.\nSingle service/concern bounded thought Similar to Arch, major issue remain separation of concerns and tradeoffs you make.\nDev’s generally tend to start by mixing all things in a single function, class, package, etc. E.g; For a web service, people tend to do transport stuff (SSL, serialization, HTTP), business logic, database handling all as single methods in single place. For non web service process, people tend to handle any communication, threading, thread coordination, configuration, business logic, etc in single place. This mixing can be seen generally in different areas as below.\nObservability: Adding anything related to observability tends to disturb business logic. E.g If you want an api metric to be present, you should be able to do that without touching BL. It generally doesn’t happen that cleanly.\nState management and access: State handling is another common thing that starts as “accessible to all” as it is the simplest thing to start with. E.g: Make all states (Tables, files, blobs, etc) accessible to all functionality. As part of architectural evolution, you start by defining clear boundaries slowly slowly in terms of modules, packages, etc.\nClass/Package issues: People would generally find it very very difficult to define boundaries of packages, classes. This is common even if classes or packages are designed with private/public functional capabilities. E.g: If a function is exposed, should it really be exposed? Is that function part of the responsibility of the class/package?\nAs newer requirements pour into the system, the architectural, service interaction and service responsibility specific tradeoffs change.\nFew examples of these changes within a service boundary level are:\nChanging levels of abstraction - a new class is created out of one big one. This may result in routing calls. Preferred way to handle this is to create a new class, let callers integrate with it, in the mean time redirect from main class to here. If the cost of maintenance turns out to be high, you have to force clients to upgrade. One middle ground here is: provide a sdk, do the rerouting in sdk, ask clients to upgrade the sdk.\nInterface change: especially if parameters are removed. This may result in building a stub to manage it. This is preferably handled via versioning. Backward incompatible changes need to upgrade major versions. Old version stays until you deprecate it. In a single codebase, modifying the callers is almost always preferred over handling rerouting, stubs, etc. Versioning is used when you don’t control the callers. Again, tradeoff is cost of modifying everybody, vs maintaining reroutes.\nSize of teams vs rules/patterns One thing that I believe is that the rules/patterns to handle change don’t really change. What changes is the tradeoffs associated with picking a solution.\nReferences Blog at a abstract level: Patterns of legacy displacement Details about patterns are present in sidebar. Critical Aggregator Divert the Flow Extract Product Lines Feature Parity Legacy Mimic Revert to Source Transitional Architecture Books that I like: Software Architecture the hard parts Clean Architecture Architectural bookshelf with different levels/context of the problems: Architect bookshelf Architect library all ",
  "date": "Oct 18, 2022",
  "tags": [
   "systems-design"
  ],
  "title": "Separation of concerns and architectural thought",
  "url": "https://pankajpipada.com/posts/2022-10-18-archlevels/"
 },
 {
  "content": "Interests For a large mount of time I used to be quite voracious in different areas of interests/hobbies. These were different things at different points of time in my life. Movies, music, books about technology and science fiction, building software systems were the main interests until now.\nFor books, the voraciousness lasted a bit longer for technology ones and to some extent for science fiction. I didn’t really get to a large number of non-tech books, but the general approach to go about it was very much similar i.e pickup, continue to the end and don’t worry about much else.\nI am not sure when I developed a taste for movies. But I watched a lot of movies. There was a stretch of time, when I used to consume upto four/five per week minimum. TV series binge watch was also present to some extent, but movies were always the primary interest. At some point I decided to keep a track of what I watch on IMDB. As of now, I have almost 1550 titles to my watched/rated list (very few were added in the last few years though).\nI sought out music from anything and anywhere. Source of discovery came from recommendation engines, suggestions from select friends whose taste I trusted, even billboard top charts for each year. This exposed me to music of different moods and flavours. My playlists were generally a mixed jumble of eras and genres.\nBuilding software systems, that could handle things that I care about, has always interested me. It could be streaming services, local movie organization methods, writing systems that could juggle a lot of activities, software helping me organize my life in certain ways are a few areas. This also resulted in me growing interested in architecture and wanting to organize software itself for organizational goals.\nStages As an undergrad, my voraciousness mainly applied to movies, books about technology, music and some literature.\nDuring my masters, literature mostly dried up, and was replaced with building and managing different technology systems. I even managed to merge two of my interests to create a small movie streaming service for on-campus consumption by students.\nDuring my first job, books of all sorts were pretty much removed. With some dis-interest in the actual job, it was actually replaced with a feeling of void.\nMy second(current) job offered great learning and experimenting opportunities w.r.t building systems. That helped in filling the void to some extent. As I grew into the job, movies dried up to some extent. Music discovery stopped and it went into a mode of listening the same known things again and again. Books about technology did pick up as a job requirement, but the voraciousness to go about it was missing.\nChange As I progressed through my career, slowly but surely, the number of things I had to take care of increased. Upto a point, I was ok with the context switches and some associated parallelism. I think, things were ok until I was able to focus on a single thing for a while, and then switch the context to a new item. Even in this mode, the voraciousness was definitely lesser, but still present to some extent. Mainly because able to focus is one thing, and being able to do it in a voracious manner is another.\nSomehow, I also grew a bit rapidly on the engineering ladder. This directly corresponds to a increase in the amount of responsibilities and additional number of things to look into. Few people specialize in depth of a topic, but my interests in building systems meant I looked at breadth a lot, plus glue work associated with it. Initially I could go into large depths too with some limited breadth, but that changed as I grew. The scale of the issue could be grasped by considering that at times I handled more than a dozen active projects along with a few in maintenance. When this started overwhelming me, I tried out a model where I was deep into a couple of projects and rest all were in a consultancy mode. As the criticality of things increased I had to put few of them under the deep involvement mode rather than consultancy.\nThe company definitely rewarded the work in different ways. Also, I did lots and lots of things that were super interesting to me. Given that I actually enjoyed things, I was able to gain back some of the voraciousness as things permitted.\nCurrent thought I am starting to believe that even though I thrive when I handle a breadth of things, it is not sustainable for my mental health and general happiness. It is irrespective of the fact that, career wise, precisely this has helped me progress. I am not really sure of the solution to this at this point, even though I am aware of it and definitely struggling with it.\nI have been told that this type of change is definitely expected as you grow in technology. Whether, I accept it and find a way to live with it or make a change for gaining back that voraciousness is upto me. I am inclined towards the later as it seems to be the only way I can imagine gaining back satisfaction and happiness, but I am not sure about lateral implications of it.\nAnother suggestion was to see if there are different hobbies that can interest me and don’t really need me to be voracious about it. Out of the multiple things I tried gardening is the only one that has stuck and genuinely interested me. It doesn’t really feel like the only thing that is going to be there, but it will definitely be one to continue for a while.\nIF I find some list of newer things, along with a few old (I wouldn’t want to let music and movies to go away from me), keeping professional multi tasking to a minimum and finding a balance between professional and personal commitments and the interests could be the way ahead. In any case, it is really hard for me to imagine getting back on a thing that I can really be voracious about. Few friends have suggested that I don’t need to have voraciousness for happiness, but the thought of it makes me really uncomfortable.\nIt is definitely clear in my mind that I need to get rid of the dryness that exist in current procrastination, movie or music explorations. All this has affected my physical fitness too. Gaining that back to some extent has also been suggested as a thing that would help me freshen up a bit, but that too seems like a large arduous as of now.\nAll in all, the present looks like an uncomfortable time for me. Writing things down generally helps me build my own clarity and I hope to get the same out of this particular write up. Hopefully I achieve this clarity sooner rather than later.\nEdit 1: Suggestion by James from HN\nGood rest, taking care of your health (physical exercise, eating well, sleeping well), and lowering your stress levels. Find what you love right now and take small steps into recultivating doing something you love doing.\n",
  "date": "Jul 9, 2022",
  "tags": [
   "life"
  ],
  "title": "Voraciousness",
  "url": "https://pankajpipada.com/posts/2022-07-09-voraciousness/"
 },
 {
  "content": "I had to setup a fresh Ubuntu dev machine after quite some time. Given that this was a loaner machine, I wanted to make sure that I have a minimal viable dev setup ready as quickly as possible.\nBelow are the steps for the minimal things I need.\nUpgrade packages For a fresh setup it is better to first make sure that everything that is upgradable is up to date. If you have a specific version requirement for any package make sure you pin it at the package manager level.\nBasic commands\n1sudo apt update 2sudo apt upgrade 3sudo apt install software-properties-common apt-transport-https wget zsh git vim tree 4sudo apt autoremove ZSH setup ZSH is an extended Bourne shell.\nTogether with Oh-My-ZSH it provides a delight full dev experience.\nI personally like to use the agnoster theme from ohmyzsh with the plugins git common-aliases zsh-syntax-highlighting zsh-autosuggestions\nFor Ubuntu terminal make sure you got to Terminal -\u003e Preferences -\u003e \u003cYour Profile\u003e -\u003e Colors and uncheck the Use system colors option so that the theme colors are used in the terminal.\nOhmyzsh standard plugins do not require explicit installation. Community plugins require some installation. Plugins links and installation guides are:\ncommon-aliases zsh-syntax-highlighting zsh-autosuggestions My simple ~/.zshrc file:\n1# If you come from bash you might have to change your $PATH. 2# export PATH=$HOME/bin:/usr/local/bin:$PATH 3 4# Path to your oh-my-zsh installation. 5export ZSH=\"/home/username/.oh-my-zsh\" 6 7ZSH_THEME=\"agnoster\" 8 9# Which plugins would you like to load? 10# Standard plugins can be found in $ZSH/plugins/ 11# Custom plugins may be added to $ZSH_CUSTOM/plugins/ 12# Example format: plugins=(rails git textmate ruby lighthouse) 13# Add wisely, as too many plugins slow down shell startup. 14plugins=(git common-aliases zsh-syntax-highlighting zsh-autosuggestions) 15 16source $ZSH/oh-my-zsh.sh 17 18export HISTSIZE=100000 19export SAVEHIST=100000 SSH key-gen and add to repositories Interacting with multiple hosted git repositories is much smoother when using SSH keys.\nSpecific git hosts provide their guides to do this, e.g: GitHub ssh key gen guide .\nGeneral setup includes:\nGenerate a ssh key pair on a machine. Add it to your git host profile settings. Test ssh access to git 1ssh-keygen -t ed25519 -C \"email@example.com\" 2ssh-keygen -t ed25519 -C \"username@example.com\" 3xclip -sel clip \u003c ~/.ssh/id_ed25519.pub 4 5# Add your git host (GitLab/GitHub/BitBucket, etc) URL for git-example.com 6ssh -T git@git-example.com Basic software install VSCode. General post for VSCode helpers is here Slack Zoom Hugo . Hugo is fantastic website building framework. Awesome for static sites. Microsoft ergonomic keyboard setup (4000 and 2019 ergonomic keyboard) All keyboard shortcuts can be configured using Settings \u003e Keyboard \u003e View and Customize shortcuts Volume keys are generally enabled by default. Recheck under Sound and media Favorite keys 1, 2 and 3 are already configured to Recent favorites of same number as Super + 1, .... To modify them to an application you can follow instructions in this StackOverflow question . tl;dr: Install dconf-editor, modify app-hotkey-x at org.gnome.shell.extensions.dash-to-dock. Reboot/Relogin after any changes via dconf-editor Media keys other than open media app (generally one that has right facing triangle inside a rectangle) can be configured using Sound and media For open media app: First disable the “Tools” key shortcut as mentioned in this StackOverflow question . tl;dr: Install dconf-editor, modify control-center-static at org.gnome.settings-daemon.plugins.media-keys. Reboot/Relogin after any changes via dconf-editor If you want to open the default media app, you can now do that using Sound and media \u003e Launch media player; If you want to set a custom application you can do that via Custom shortcuts \u003e + \u003e {name; e.g music}, {your music apps cli command ; e.g: youtube-music}, {Tools} For Calculator you can set it in Launchers. Snipping is equivalent to PrtScn. App switcher is equivalent to Activities or Tab switcher Search is equivalent to Activities i.e (windows button) Emoji needs to be set as a Custom shortcut \u003e + \u003e {Characters}, {gnome-characters}, {press the emoji button. It will come as Shift + Ctrl + Alt + Super + Space}. Office button can be set similarly to whatever you want. Programming language settings Go. Basic introductory primer is present here . Python. Getting started link Pull your code and go exploring :)\n",
  "date": "Jul 24, 2020",
  "tags": [
   "linux-utils"
  ],
  "title": "Linux - Ubuntu initial dev setup",
  "url": "https://pankajpipada.com/posts/2020-07-24-ubuntu-setup/"
 },
 {
  "content": "Below are the steps recommended to read the Paxos made simple paper by Leslie Lamport and understand Paxos.\nSteps Read full paper. Paxos made simple “Begin at the beginning,” the King said gravely, “and go on till you come to the end: then stop.” -- The King, Alice in wonderland. Re-look at why is “leader election required”. 2.5 -\u003e 2.4 -\u003e 2.3\nRe-look the requirements for proposer P2c -\u003e P2b -\u003e P2a -\u003e P2\nRe-look the requirements for acceptor P1a -\u003e P1\nGo through the algorithm again. i.e Phase 1 and Phase 2.\nRepeat 2-4 again. Do 1-4, if still not sure.\n",
  "date": "May 25, 2020",
  "tags": [
   "papers"
  ],
  "title": "Understanding Paxos",
  "url": "https://pankajpipada.com/posts/2020-05-25-understanding-paxos/"
 },
 {
  "content": "Docsify, YAML front-matter, mustache templates \u0026 tags and some quirks when using them.\nDocsify Docsify is a great documentation site generator.\nIt generates your documentation website on the fly using Markdown files directly. To start using it, all you need to do is create an index.html and deploy it on GitHub pages or any other static site host. It has a great plugin system that enables extensibility and can be used for solving multiple use cases. Docsify-Mustache One such plugin that is greatly useful is Docsify-Mustache .\nIt allows preprocessing markdown documents with Mustache template engine. Mustache is a logic-less templating system. It works by expanding tags in a provided template using values provided in a hash or object. E.g: If you use {{name}} as template in your markdown and provide the value for name either via YAML front-matter or any other supported sources , that value will get rendered. How to use this plugin with docsify is very well explained in the documentation site for this plugin. Mustache tags Mustache supports different types of tags as documented in mustache manual . The basic tags, that were useful to me, when dealing with a documentation site are Variables, Sections with non-empty lists and Inverted sections. Below are a few examples on using these mustache tag types. I have considered the source of the “values” as YAML front-matter in markdown but as pointed out before, the source can be any supported input type. Variables Variables are the most basic tag type in mustache. The template for accessing a variable is {{variable_name}}.\nExample:\nConsider that you declare the following YAML front-matter in your markdown document:\n1--- 2title: My awesome project documentation 3category: Useful 4--- Now, if you want to refer this in the markdown file you can add:\n1{{title}} 2 3This project belongs to category: {{category}} When rendering the page title and category will be substituted using the values declared in the front matter.\nSections with non-empty lists Sections render blocks of text one or more times, depending on the value of the key in the current context.\nA section begins with a pound and ends with a slash. That is, {{#person}} begins a \"person\" section while {{/person}} ends it.\nWhen the value for a section is a non-empty list, the text in the block will be displayed once for each item in the list. The context of the block will be set to the current item for each iteration. In this way we can loop over collections.\nIMPORTANT NOTE: If the list values contain a hyphen - in them then the list rendering is incorrect/fails for that value. To avoid this use single quotes (’’) around the value as depicted below.\nExample:\nConsider that you declare the following YAML front-matter in your markdown document:\n1--- 2tags: [useful, \"rocket-science\", launch] 3--- Now, if you want to refer the tags list in the markdown file you can add:\n1This doc is has the following tags: 2{{#tags}} 3{{.}} 4{{/tags}} Output will be rendered as:\n1This doc is has the following tags: useful, rocket-science, launch Inverted sections Inverted sections may render text once based on the inverse value of the key. That is, they will be rendered if the key doesn’t exist, is false, or is an empty list.\nAn inverted section begins with a caret (hat) and ends with a slash. That is {{^person}} begins a “person” inverted section while {{/person}} ends it.\nExample:\nConsider that you declare the following YAML front-matter in your markdown document:\n1--- 2tags: [] 3--- Now, if you want to handle the tags list being empty and check for category being absent you can add:\n1This doc is has the following tags: 2{{^category}} 3No category found !!! 4{{/category}} 5 6{{^tags}} 7No tags found !!! 8{{/tags}} Given that the categories key doesn’t exist and the tags list is empty, Output will be rendered as:\n1No categories found !!! 2No tags found !!! Extended example An extended example where we add a YAML front-matter to a markdown file, use the variables and handle the absent cases is given below.\nMarkdown file that adds a constant heading section to the documentation page, where the title will be displayed first, then category will be displayed and then the tags list is provided:\n1--- 2title: My awesome project documentation 3tags: [useful, \"rocket-science\", launch] 4--- 5 6# {{title}} 7 8{{category}} 9{{^category}} 10No category found !!! 11{{/category}} 12 13Tag list: 14{{#tags}} 15{{.}} 16{{/tags}} 17{{^tags}} 18No tags found !!! 19{{/tags}} The output rendered will be:\n1My awesome project documentation 2 3No category found !!! 4 5Tag list: useful, rocket-science, launch ",
  "date": "Apr 29, 2020",
  "tags": [
   "markdown"
  ],
  "title": "Mustache templates and YAML front-matter with Docsify",
  "url": "https://pankajpipada.com/posts/2020-04-29-docsify-mustache/"
 },
 {
  "content": "Basic shortcuts, useful plugins, and sample settings file for VSCode.\nThis is a short list that should help anyone to get started with VSCode. It does not go into advanced mode or doesn’t serve as a long “cheat sheet”. Intention is to help in getting started rather than doing advanced stuff.\nBasic shortcuts Description Mac Linux New file cmd(⌘) n ctrl n Search in file cmd(⌘) f Ctrl f Search across files cmd(⌘) shift(⇧) f ctrl shift f Go to Definition f12 f12 Go back ctrl(⌃) - ctrl - Go to file cmd(⌘) p ctrl p Open command pallette cmd(⌘) shift(⇧) p ctrl shift p Replace in file cmd(⌘) option(⌥) f ctrl alt f Replace across files cmd(⌘) shift(⇧) h ctrl shift h Plugins Name Description GitLens Git supercharged Python Full python support. Lint, debug, intellisense, format, etc Go Full Go support. Lint, debug, intellisense, format, etc Markdown All in One All you need to write Markdown. Keyboard shortcuts, table of contents, auto preview and more Markdownlint Markdown lint and style check Code spell checker Spelling checker for source code Prettier Prettier/Formatter for multiple formats Regex Description regex Search any character or new line `((. Group things and access in replace Add things in ( and ) and access as $1,… ",
  "date": "Apr 28, 2020",
  "tags": [
   "linux-utils"
  ],
  "title": "VSCode basic helpers",
  "url": "https://pankajpipada.com/posts/2020-04-28-vscode/"
 },
 {
  "content": "Few git commands.\nDescription Command Delete all branches locally except for ones having the word “master” `git branch Pull submodules initially git submodule update --init --recursive Update submodules git submodule update --recursive --remote Clone using username pass in URL `git clone http://${GIT_USERNAME}:$(echo -n $GIT_PASSWORD ",
  "date": "Mar 27, 2020",
  "tags": [
   "linux-utils"
  ],
  "title": "Linux - Git commands",
  "url": "https://pankajpipada.com/posts/2020-03-27-git-commands/"
 },
 {
  "content": "Generic references and points to consider when doing a memory analysis with go programming language.\nDo pprof memory profiling .\nGo returns memory to OS gradually. Typical time is ~5 minutes. If immediate return is needed, we could use FreeOSMemory . General recommendation is that if this is needed, manage memory alternatively.\nAlternative for frequent allocated and reused objects is sync.Pool usage. This is tricky and have to be careful when doing this. Sample usage can be found in blog .\nExcellant High performance go tips are avilable at this post . This talks about memory and GC too.\nGarbage collection behaviour is well explianed at: Ardan labs Post 1 , Post 2 and Post 3 . Also there is this keynote explaining the evolution of Go’s garbage collector.\n",
  "date": "Mar 24, 2020",
  "tags": [
   "golang"
  ],
  "title": "Memory analysis in Go",
  "url": "https://pankajpipada.com/posts/2020-03-24-go-memory-profiling/"
 },
 {
  "content": "Organizing a local storage based movie collection. At a high level it involves:\nPrepare movie metadata Renaming and folder arrangement Manage movies in Kodi Prepare movie metadata Use TinyMediaManager i.e tmm for metadata management\nStep 1: Prepare source folders:\nImport local movie paths These are called as sources in tmm. Add these via settings Clean duplicates Manage multi file movies Naming convention is important. Files should end with -cd1, -cd2, etc or -part1, -part2, etc. Bulk Subtitles find and download is generally a bad idea. Lot of inaccuracies in downloads. Better do this one by one and before any renaming of files. Step 2: Scrape metadata\nGenerally prefer Kodi format metadata (nfo) files. tmm provides option for this in the setting. IMDb vs TMDb metadata IMDb scrapping generally is slow for bulk operations TMDb provides a good source for movie metadata and most of it (including fan art and posters) can be downloaded from here. I prefer rating from IMDb. There is an option to select just ratings from IMDb in tmm I would recommend doing a two pass metadata search In the first pass do a bulk operation using TMDb scrapper. This can include TMDb ratings. Once all metadata is downloaded a second bulk search can be done just for ratings and top 250 data using IMDb. Renaming and folder arrangement This step move movies to a Kodi recommended structure using tmm. CAUTION Once renaming is complete it cannot be undone. Verify each and every thing before doing renaming. Recommended folder structure is movie-name (year). Generally prefer a flat structure over sub-directories and deep hierarchies. General preference for file names is same as folder name. I personally prefer movie-name (year) video-resolution imdb-rating part-no. Once folder name and file name preferences are set in Settings-\u003eMovies-\u003eRenamer, do a dry run to see the changes that tmm is going to do. Verify the dry run result carefully. Adjust settings if something seems weird. Execute rename once verified. Manage movies in Kodi Install Kodi on your laptop/desktop/tv. Import the media folder created using tmm above in Kodi media manager. As a general setting enable Update library on startup. Cleaning DB in Kodi is a bit of a pain. Avoid folder/movie path changes. Keep running clean library even for smaller changes done to files/folders. Periodically i.e once you have watched enough movies you may want to do a export library. This writes watched status and any extra info in Kodi to .nfo files in source. ",
  "date": "Mar 17, 2020",
  "tags": [
   "linux-utils",
   "movie"
  ],
  "title": "Movies - Organizing a largish movie collection",
  "url": "https://pankajpipada.com/posts/2020-03-17-organizing/"
 },
 {
  "content": " Lifes irreducible structure Paxos made simple The Google file system - GFS Amazon DynamoDB at USENIX ATC 22 Dynamo: Amazon’s Highly Available Key-value Store MapReduce: Simplified Data Processing on Large Clusters TAO: Facebook’s Distributed Data Store for the Social Graph Dapper, a Large-Scale Distributed Systems Tracing Infrastructure Bigtable: A Distributed Storage System for Structured Data Papers we love github repo Edit: 17 December 2024\nEmotional Design by Donald Norman ",
  "date": "Mar 12, 2020",
  "tags": [
   "papers"
  ],
  "title": "Awesome papers list",
  "url": "https://pankajpipada.com/posts/2020-03-12-awesome-papers/"
 },
 {
  "content": "Steps to set up a samba server on ubuntu 18.04.\n1sudo apt update 2sudo apt install samba 3# Allow Samba in ufw firewall 4sudo ufw allow 'Samba' 5sudo systemctl status smbd 6# Create a directory to host Samba share 7sudo mkdir /disk1/samba 8 9#### User setup 10sudo useradd -M -d /disk1/samba/peewee -s /usr/sbin/nologin -G sambashare peewee 11sudo mkdir /disk1/samba/peewee 12sudo chown peewee:sambashare /disk1/samba/peewee 13sudo chmod 2770 /disk1/samba/peewee 14sudo smbpasswd -a peewee # set password here 15sudo smbpasswd -e peewee 16vi /etc/samba/smb.conf 17 ## Add these to the globals section to avoid name mangling and using appropriate charset 18 # [globals] 19 mangled names = no 20 dos charset = CP850 21 unix charset = UTF-8 22 [peewee] 23 path = /disk1/samba/peewee 24 browseable = yes 25 read only = no 26 force create mode = 0660 27 force directory mode = 2770 28 valid users = peewee 29 30sudo systemctl restart smbd 31sudo systemctl restart nmbd ",
  "date": "Jan 30, 2020",
  "tags": [
   "linux-utils"
  ],
  "title": "Samba setup",
  "url": "https://pankajpipada.com/posts/2020-01-30-samba-setup/"
 },
 {
  "content": "Few ubuntu server setup issues and corresponding steps to debug them.\nDisabling floppy drive: Error: print_req_error: I/O error, dev fd0, sector 0\n1sudo rmmod floppy 2sudo echo \"blacklist floppy\" | sudo tee /etc/modprobe.d/blacklist-floppy.conf 3sudo dpkg-reconfigure initramfs-tools Error: Network not up: eth/ens doesnt show or no service network-manager\n1sudo dhclient 2sudo apt update 3sudo apt install network-manager ifupdown 4sudo service network-manager restart 5sudo systemctl status NetworkManager.service 6 7sudo vi /etc/netplan/01-netcfg.yaml 8 # This file describes the network interfaces available on your system 9 # For more information, see netplan(5). 10 network: 11 version: 2 12 renderer: NetworkManager 13 ethernets: 14 ens32: 15 dhcp4: yes 16sudo netplan generate 17sudo netplan apply ssh setup\n1## For regenerating only rsa key 2sudo ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key 3## For regenerating all missing keys 4sudo ssh-keygen 5# service ssh restart 6sudo systemctl status ssh Fix non-configured locales\n1sudo locale-gen en_US en_US.UTF-8 2sudo dpkg-reconfigure locales ",
  "date": "Jan 30, 2020",
  "tags": [
   "linux-utils"
  ],
  "title": "Linux - Ubuntu 18.04 server setup debug",
  "url": "https://pankajpipada.com/posts/2020-01-30-ubuntu-1804-server-debug/"
 },
 {
  "content": "Generic references to get started with go programming language.\nGetting started links Installation How to Write Go code Tour of Go Effective Go Reference documentation Project layout Package Oriented Design Style guide Standard project layout Naming conventions Standard Package names go blog Go Talk Few important rules: Package names are singular, short, clear, lower case, with NO under_scores or mixedCaps. Package content: Avoid stutter, simplify function names. Avoid meaningless package names such as util, lib, common, or misc. Function and variable names: The convention in Go is to use MixedCaps or mixedCaps rather than underscores to write multiword names. Keep local variables short. Common variable/type combinations can use really short names. E.g: ‘i’ - for index, ‘r’ for reader, ‘b’ for buffer. Acronyms should be all capitals: E.g: ServeHTTP and IDProcessor Validation: Use gofmt for autoformatting your code. Use golint . It would provide warnings related to naming and styling of go code. Code comments Standard Go Blog Few important rules: The convention is simple: to document a type, variable, constant, function, or even a package, write a regular comment directly preceding its declaration, with no intervening blank line. Godoc will then present that comment as text alongside the item it documents. The comment is a complete sentence that begins with the name of the element it describes. Comments on package declarations should provide general package documentation. Use doc.go for packages that need large amount of introductory documentation. Swagger documentation for APIs: Can use code annotations as described at GoSwagger Generic guidelines Writing clear idiomatic Go code: Effective Go Few important rules: Dependencies should be passed explicitly. E.g: Pass logger explicitly. Do NOT keep unused functions, constants, variables, types. Do NOT use panics/recover as exception catching mechanism. Avoid blanket error handling. Avoid Misusing errors. Avoid long functions. A function should not exhibit split personality. Avoid global objects. Validation: Static check Race detector Vet CI: GolangCI-Lint Testing Guidelines: Go provides command go test for running tests in \\*\\_test.go files. It also has support for benchmarking. go test has support for race detector using - race. Use it while running tests. Test file containing component tests should specify the build constraints so that files can be identifiable whether it has Unit or Component or Integration test. The build constraints will be helpful to exclusively run only UT or Component tests or Integration tests. Command “go test ./… -tags UT” will only run unit test file and will exclude file component_test.go as it as define build constraint “!UT” E.g File component_test.go // +build !UT … some tests Tools: Package for boiler plate test code generation: gotests This is available as a plugin in VSCode and other major IDEs. Package for generating mocks for interfaces: mock Package for mocking SQL: go-sqlmock Test helpers: testing Measuring code coverage Go test can be configured -cover to collect code coverage information. It does not have support to find code coverage in workflow testing. But can be tweaked to collect coverage information by writing test for entry point function of binary. ",
  "date": "Nov 17, 2019",
  "tags": [
   "golang"
  ],
  "title": "Go introductory primer",
  "url": "https://pankajpipada.com/posts/2019-11-17-go-intro-primer/"
 },
 {
  "content": "Basic docker commands.\nCommands Description Command List all container instances, with their ID and status docker ps -a Lists all images on the local machine docker images Displays the logs from a running container docker logs [container name or ID] Stop all containers docker stop $(docker ps -a -q) Delete all containers docker rm $(docker ps -a -q) Changes command prompt from the host to a running container docker attach [container name or ID] Executes a command within a running container docker exec [container name or ID] shell command ",
  "date": "Nov 11, 2019",
  "tags": [
   "linux-utils"
  ],
  "title": "Docker - Commands",
  "url": "https://pankajpipada.com/posts/2019-11-11-docker-commands/"
 },
 {
  "content": "Random collection of commands for Ubuntu Linux.\nTop Description Command display top with threads top -H top with output sorted by memory top -o %MEM run top in batch mode 10 times with 5 seconds delay in command mode with output sorted by memory top -b -n 10 -d 5 -c -o %MEM run top in batch mode 10 times with 5 seconds delay in command mode with output sorted by memory and only print 15 lines at a time `top -b -n 10 -d 5 -c -o %MEM Jekyll Reference - Using with bundler Description Command serve jekyll locally bundle exec jekyll serve Virtualbox Description Command create a vmdk using raw device VBoxManage internalcommands createrawvmdk -filename \"\u003c/path/to/file\u003e.vmdk\" -rawdisk /dev/sdb Rsync Description Command rsync to a remote server using ssh protocol and show progress rsync -avzhe ssh --progress ./localfolder user@\u003cremote server name/ip\u003e:/remote/folder/location Option to transfer files upto a certain size --max-size=\u003cn\u003em Disk paritions Create a ntfs partition from an empty disk\n1sudo fdisk /dev/sdb 2# fdisk is interactive 3# press m for help 4# Press p to list any available partitions 5# create a new partition by using n 6# after altering partitions press w to write 7# make a ntfs file system with \"quick format\" i.e dont write zeroes and dont check for bad sectors 8# remove f for full format 9mkfs.ntfs -f /dev/sdb1 10blkid 11# Note the UUID of the partition E.g: /dev/sdb1 UUID=\"asdfg1246\" 12# Adding a entry in /etc/fstab 13 UUID=asdfg1246 /disk1 ntfs-3g permissions,locale=en_US.utf8 0 2 ",
  "date": "Nov 10, 2019",
  "tags": [
   "linux-utils"
  ],
  "title": "Linux - Ubuntu - Random commands",
  "url": "https://pankajpipada.com/posts/2019-11-10-commands/"
 },
 {
  "content": "References, basic commands and sample rc file for GNU screen.\nReferences Man Page Quick Reference Basic commands Description Command Start a new session with session name screen -S \u003csession_name\u003e List running sessions / screens screen -ls Attach to a running session with name screen -R \u003csession_name\u003e Detach a running session screen -d \u003csession_name\u003e Command mode Ctrl+a Enable vertical scrolling mode in a running session Ctrl-a ESC Create new window Ctrl-a c Change to window by number Ctrl-a \u003cnumber\u003e Enter screen command Ctrl-a : Send command to the screen session screen -X -S \u003csession_name\u003e \u003ccommand\u003e Send kill command to the screen session screen -X -S \u003csession_name\u003e kill RC File 1# location: ~/.screenrc 2# A sample screenrc file with a hardstatus line at bottom 3# 3 windows created, with custom commands stuffed into each at session creation 4# and some key bindings. 5 6# the following lines give a status line, with the current window highlighted 7hardstatus alwayslastline 8hardstatus string '%{= kg}[%{G}%H%? %1`%?%{g}][%= %{= kB}%?%-Lw%?%{+b r}(%{G}%n*%f %t%?(%u)%?%{r})%{-b B}%?%+Lw%?%?%= %{g}%][%{B}%d/%m %{W}%C%A%{g}]' 9#hardstatus string '%{= kg}[ %{G}%H %{g}][%= %{= kB}%?%-Lw%?%{+b r}(%{G}%n*%f %t%?(%u)%?%{r})%{-b B}%?%+Lw%?%?%= %{g}%]' 10#hardstatus string '%{= kG}[%{G}%H%? %1`%?%{g}][%= %{= kw}%-w%{+b yk} %n*%t%?(%u)%? %{-}%+w %=%{g}][%{B}%d/%m %{W}%C%A%{g}]' 11 12# huge scrollback buffer 13defscrollback 10000 14 15# no welcome message 16startup_message off 17 18# 256 colors 19# attrcolor b \".I\" 20# termcapinfo xterm 'Co#256:AB=\\E[48;5;%dm:AF=\\E[38;5;%dm' 21defbce on 22 23# mouse tracking allows to switch region focus by clicking 24# mousetrack on 25# default windows 26screen -t HOME 0 bash 27stuff \"cd /root\" 28screen -t SRV 1 bash 29stuff \"cd /root/srv\" 30screen -t MYSQL 2 bash 31stuff \"cd /root/mysql\" 32select 0 33#bind c screen 1 # window numbering starts at 1 not 0 34#bind 0 select 10 35 36# get rid of silly xoff stuff 37#bind s split 38 39# navigating regions with Ctrl-arrows 40bindkey \"^[[1;5D\" focus left 41bindkey \"^[[1;5C\" focus right 42bindkey \"^[[1;5A\" focus up 43bindkey \"^[[1;5B\" focus down 44 45# switch windows with F3 (prev) and F4 (next) 46bindkey \"^[OR\" prev 47bindkey \"^[OS\" next 48 49# switch layouts with Ctrl+F3 (prev layout) and Ctrl+F4 (next) 50bindkey \"^[O1;5R\" layout prev 51bindkey \"^[O1;5S\" layout next 52 53# F2 puts Screen into resize mode. Resize regions using hjkl keys. 54bindkey \"^[OQ\" eval \"command -c rsz\" # enter resize mode 55 1,1 Top 56# switch layouts with Ctrl+F3 (prev layout) and Ctrl+F4 (next) 57bindkey \"^[O1;5R\" layout prev 58bindkey \"^[O1;5S\" layout next 59 60# F2 puts Screen into resize mode. Resize regions using hjkl keys. 61bindkey \"^[OQ\" eval \"command -c rsz\" # enter resize mode 62 63# use hjkl keys to resize regions 64bind -c rsz h eval \"resize -h -5\" \"command -c rsz\" 65bind -c rsz j eval \"resize -v -5\" \"command -c rsz\" 66bind -c rsz k eval \"resize -v +5\" \"command -c rsz\" 67bind -c rsz l eval \"resize -h +5\" \"command -c rsz\" 68 69# quickly switch between regions using tab and arrows 70bind -c rsz \\t eval \"focus\" \"command -c rsz\" # Tab 71bind -c rsz -k kl eval \"focus left\" \"command -c rsz\" # Left 72bind -c rsz -k kr eval \"focus right\" \"command -c rsz\" # Right 73bind -c rsz -k ku eval \"focus up\" \"command -c rsz\" # Up 74bind -c rsz -k kd eval \"focus down\" \"command -c rsz\" # Down 75#source .screen_layout 76#layout save def ",
  "date": "Nov 9, 2019",
  "tags": [
   "linux-utils"
  ],
  "title": "GNU Screen helpers",
  "url": "https://pankajpipada.com/posts/2019-11-09-screen/"
 },
 {
  "content": "A script to chain multiple closures in python with example usage.\nClosure chaining 1from functools import wraps 2 3 4# A new closure will be returned as a chain of Closures in the incoming list order 5def chain_closures(closure_list=None): 6 if not closure_list: 7 return None 8 9 def chain(f): 10 @wraps(f) 11 def r(*args, **kwargs): 12 newfunc = f 13 for c in reversed(closure_list): 14 newfunc = c(newfunc) 15 return newfunc(*args, **kwargs) 16 17 return r 18 19 return chain Example 1from functools import wraps 2from closures import chain_closures 3import logging 4 5logger = logging.getLogger() 6 7def x(a): 8 def request_logger(f): 9 @wraps(f) 10 def rlog(*args, **kwargs): 11 logger.info(\"%s x entry\", a) 12 ret = f(*args, **kwargs) 13 logger.info(\"%s x exit\", a) 14 return ret 15 16 return rlog 17 18 return request_logger 19 20 21def y(b): 22 def request_logger(f): 23 @wraps(f) 24 def rlog(*args, **kwargs): 25 logger.info(\"%s y entry\", b) 26 ret = f(*args, **kwargs) 27 logger.info(\"%s y exit\", b) 28 return ret 29 30 return rlog 31 32 return request_logger 33 34 35def z(a, b): 36 c1 = x(a) 37 c2 = y(b) 38 39 def request_logger(f): 40 @wraps(f) 41 def rlog(*args, **kwargs): 42 q = c1(c2(f)) 43 logger.info(\"z entry\") 44 ret = q(*args, **kwargs) 45 logger.info(\"z exit\") 46 return ret 47 48 return rlog 49 50 return request_logger 51 52 53def w(a, b, f): 54 c1 = x(a) 55 c2 = y(b) 56 q = c1(c2(f)) 57 return q 58 59 60def get_w(a, b): 61 c1 = x(a) 62 c2 = y(b) 63 q = chain_closures([c1, c2]) 64 return q 65 66 67def f1(a, b): 68 logger.info(\"%s %s\", a, b) 69 # raise Exception(\"me except\") 70 71 72def smain(): 73 c4 = get_w(10, 11) 74 f2 = c4(f1) 75 f2(12, 13) 76 logger.info(f2.__name__) 77 78 79if __name__ == \"__main__\": 80 logging.basicConfig(format='%(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S', level=logging.DEBUG) 81 smain() ",
  "date": "Nov 9, 2019",
  "tags": [
   "python"
  ],
  "title": "Python Helpers - Closures",
  "url": "https://pankajpipada.com/posts/2019-11-09-py-helpers-closures/"
 },
 {
  "content": "A script to cprofile a single function in python.\nCProfile a single function in python 1import logging 2import cProfile 3import pstats 4import StringIO 5 6from closure_chaining_example import smain 7 8logging.basicConfig(format='%(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S', level=logging.DEBUG) 9pr = cProfile.Profile() 10pr.enable() 11smain() 12# do something 13pr.disable() 14s = StringIO.StringIO() 15sortby = 'cumulative' 16ps = pstats.Stats(pr, stream=s).sort_stats(sortby) 17ps.print_stats() 18logging.info(\"Profilestats: %s\", s.getvalue()) ",
  "date": "Nov 9, 2019",
  "tags": [
   "python"
  ],
  "title": "Python Helpers - Profiling",
  "url": "https://pankajpipada.com/posts/2019-11-09-py-helpers-profiling/"
 }
]