-
-
Notifications
You must be signed in to change notification settings - Fork 6
YouTube Tutorial
Link to Video: https://www.youtube.com/watch?v=7pDfyqBZwvo
Let’s take a look at web search in various browsers
allows using different searhc engines and defining shortcuts
allows using different searhc engines and defining shortcuts
- allows using different searhc engines and defining shortcuts
- has chatgpt and gmail as built-in search engines
- allows doing instant open (simialr to Google’s I’m feeling Lucky)
- Force you to pick a default search engine (instead of showing results from multiple sources)
- Not much control over backends (e.g. which GenAI to use)
- Limited control over UI, information presentation, sorting, …
- Accessing previous search is not straightforward (you have to rely on suggestions or open history yourself)
- Search results are not neccessarily reproducibile and there is no easy way to save the results
Let’s take a look at current available packages in Emacs.
a minimal browser, mainly useful for plain text, but does not really add much new functionality to searching.
A webkit (a.k.a. Safari) built-in in emacs. Everything else is the same as a normal browser (perhaps with just less security)
Webjump - WikEmacs is a simple built-in command that sends a query to search engines.
(setq webjump-sites
`(("oogle" .
[simple-query "www.google.com" "www.google.com/search?q=" ""])
("DuckDuckGo" .
[simple-query "duckduckgo.com" "duckduckgo.com/?q=" ""])
(" Wikipedia" .
[simple-query "wikipedia.org" "wikipedia.org/wiki/" ""])
(" StackOverflow" .
[simple-query "www.stackoverflow.com"
"www.stackoverflow.com/search?q="
""])
(" Emacs Wiki" .
[simple-query "www.emacswiki.org" "www.emacswiki.org/cgi-bin/wiki/" ""])
(" github" .
[simple-query "www.github.com" "https://github.com/search?q=" ""])
(" Homepage"
[simple-query "www.armindarvish.com" "https://www.armindarvish.com/q=" ""])
))engine-mode is similar to webjump, but with more bells and whisltles.
(use-package engine-mode
:straight t
:config
(defengine duckduckgo
"https://duckduckgo.com/?q=%s"
:term-transformation-hook upcase)
(engine-mode t))emacs-google-this is fesigned for searching thing at point
(use-package google-this
:straight t)emacs-helm · GitHub provides Google autosuggestion inside emacs.
counsel-web provides Google Search Results directly inside Emacs minibuffer, but because it uses HTML parsing your IP may get flagged if you hit the server too frequently.
For details, refer to the documentation on the Github repo here: consult-web. Maker sure to adjust the API key values for your own keys.
(use-package consult-web
:straight (consult-web :type git :host github :repo "armindarvish/consult-web" :branch "develop" :files (:defaults "extras/*.el" "sources/*.el"))
:after consult request elfeed
:custom
(consult-web-show-preview t)
(consult-web-preview-key "C-o")
(consult-web-highlight-matches t)
(consult-web-dynamic-input-debounce 0.8)
(consult-web-dynamic-input-throttle 1.6)
(consult-web-dynamic-refresh-delay 0.8)
:config
(require 'consult-web-sources)
(require 'consult-web-embark)
;; set api keys
(setq consult-web-google-customsearch-key "YOUR-GOOGLE-API-KEY"
consult-web-google-customsearch-cx "YOUR-GOOGLE-CX-NUMBER"
consult-web-brave-api-key "YOUR-BRAVE-API-KEY"
consult-web-brave-autosuggest-api-key "YOUR-BRAVE-AUTOSUGGEST-API-KEY"
consult-web-openai-api-key "YOUR-OPENAI-API-KEY"
consult-web-stackexchange-api-key "YOUR-STACKEXCHANGE-API-KEY"
consult-web-pubmed-api-key "YOUR-PUBMED-API-KEY"
consult-web-bing-search-api-key "YOUR-BING-API-KEY"
consult-web-scopus-api-key "YOUR-SCOPUS-API-KEY"
)
)We can do google search by calling consult-web-google.
(setq consult-web-default-autosuggest-command #'consult-web-dynamic-brave-autosuggest)(setq consult-web-default-count 10)now let’s use Brave Search API by calling consult-web-brave
(setq consult-web-group-by nil)(defun ad/consult-web-group-test (cand transform)
(when-let* ((url (get-text-property 0 :url cand))
(urlobj (if url (url-generic-parse-url url) nil))
(domain (if (url-p urlobj) (url-domain urlobj)))
(name (cond
((string-match ".*\\.gov.*\\|.*\\.uk.*\\|.*\\.eu.*" domain nil nil)
"Government")
((string-match ".*\\.edu.*" domain nil nil)
"University")
((member domain '("github.com"
"gitlab.com"
"bitbucket.org"
"sourceforge.net"
"sourcehut.org"
"codeberg.org"
))
"Code")
((member domain '("wikipedia.org"
"dictionary.com"
"merriam-webster.com"
"britannica.com"))
"Encyclopedia")
((member domain '("reddit.com"
"stackexchange.com"
"stackoverflow.com"
"quora.com"))
"Community")
((member domain '("orgmode.org"
"gnu.org"
"fsf.org"
"emacswiki.org"
"masteringemacs.org"))
"FOSS")
((member domain '("spotify.com"
"pandora.com"))
"Music")
((member domain '("youtube.com"
"vimeo.com"))
"Videos")
((member domain '("nytimes.com"
"washingtonpost.com"
"cnn.com"
"bbc.com"
"economist.com"
"apnews.com"
"theguardian.com"))
"News")
(t
"Others"))))
(if transform (substring cand) name)))
(setq consult-web-group-by #'ad/consult-web-group-test)(setq consult-web-group-by :domain)use Emacs Web Wowser
(setq consult-web-default-browse-function #'eww-browse-url)use org-web-tools
(setq consult-web-default-browse-function #'org-web-tools-read-url-as-org)(setq ad/browsers
(cl-remove nil `(,(if (functionp 'browse-url-arc) (cons "Arc" #'browse-url-arc))
,(if (functionp 'browse-url-firefox) (cons "Firefox" #'browse-url-firefox))
,(if (functionp 'browse-url-chrome) (cons "Chrome" #'browse-url-chrome))
,(if (functionp 'eww-browse-url) (cons "EWW" #'eww-browse-url))
,(if (functionp 'org-web-tools-read-url-as-org) (cons "Org-Web-Tools" #'org-web-tools-read-url-as-org))
,(if (functionp 'browse-url-nyxt) (cons "Nyxt" #'browse-url-nyxt)))))
(defun ad/browse-url (&rest args)
"Select the prefered browser from a menu before opening the URL."
(interactive)
(let ((browser (consult--read my:browsers
:prompt "Choose Browser: "
:require-match t))
(args (or args (browse-url-interactive-arg "URL: "))))
(apply (cdr (assoc browser my:browsers)) args)))
(setq consult-web-default-browse-function #'ad/browse-url)we can use consult-web-chatgpt
We cna use the package gptel for a full-feature AI chat.
Install gptel following the official documentation and then use consult-web-gptel.
Let’s set the sources we want ot use:
(setq consult-web-default-count 5)
(setq consult-web-multi-sources '("gptel"
"Brave"
"Wikipedia"
"StackOverflow"
"YouTube"))Now you can run the interactive command consult-web-multi
Let’s try consult-web-dynamic-google
Let’s see how passing arguments is useful for gptel consult-web-dynamic-gptel
Try entering the following in the minibuffer search (if you have acces to gpt-4!)
how to do web search in emacs -- --model gpt-4
Let’ set the sources for multi-source dynamic search:
(setq consult-web-dynamic-sources '("gptel" "Brave" "Wikipedia" "StackOverflow" "YouTube"))Now try the interactive command consult-web-dynamic.
Use consult-web-dynamic-consult-line-multi to see the consult-web version of consult-line-multi.
You need to install and setup consult-notes. Then try using consult-web-zettel-roam-nodes or the equivalent in your settings (the name of the function might be different depending on your consult-notes config).
If you use elfeed, then try consult-web-dynamic-elfeed, otherwise first install and set up elfeed and then try the command.
Now let’s set sources for a dynamic multi-source “Omni” search (meaning both local and web sources)
(setq consult-web-dynamic-omni-sources (list "Known Project" "File" "Bookmark" "Buffer" "Reference Roam Nodes" "Zettel Roam Nodes" "Line Multi" "elfeed" "Brave" "Wikipedia" "gptel" "Youtube"))Then try the interactive command consult-web-dynamic-omni!
Searching Academic Literature
Set up your pubmed account and API key. Then try the interactive command consult-web-dynamic-pubmed
Set sources for multi-source academic literature search:
(setq consult-web-scholar-sources '("PubMed" "Scopus"))Then try consult-web-scholar.
chnage the preview function to EWW (GNU Emacs Manual) ( or org-web-tools if available) and try reading an article
(setq consult-web-default-preview-function #'eww-browse-url)Here is an example for taking notes on academic literature. You can further connect this to org-capture templates as well.
(defun consult-web-embark-scholar-insert-note (cand)
"insert note snippet for article"
(let* ((url (and (stringp cand) (get-text-property 0 :url cand)))
(url (and (stringp url) (string-trim url)))
(doi (and (stringp cand) (get-text-property 0 :doi cand)))
(doi (if (and doi (stringp doi)) (concat "https://doi.org/" doi)))
(source (and (stringp cand) (get-text-property 0 :source cand)))
(url (if (and (equal source "Scopus") doi)
doi
url))
(title (and (stringp cand) (get-text-property 0 :title cand)))
(authors (and (stringp cand) (get-text-property 0 :authors cand)))
(authors (cond
((and (listp authors) (= (length authors) 1))
(car authors))
((listp authors)
(mapconcat #'identity authors ", "))
(t authors)))
(journal (and (stringp cand) (get-text-property 0 :journal cand)))
(date (and (stringp cand) (get-text-property 0 :date cand))))
(cond
((derived-mode-p 'org-mode)
(insert (concat
"\n"
(cond
((and url title) (format "** [[%s][%s]]\n" url title))
(url (format "** [[%s]]\n" url))
(title (format "** %s\n" title)))
(if authors (format "\n%s" authors))
(if journal (format "\nin =%s= " journal))
(if date (format "published on [%s]\n" date) "\n")
"\n*** Notes\n"
)))
((derived-mode-p 'markdown-mode)
(insert (concat
"\n"
(cond
((and url title) (format "## [%s](%s)\n" url title))
(url (format "## <%s>\n" url))
(title (format "## %s\n" title)))
(if authors (format "\n%s" authors))
(if journal (format "\nin **%s** " journal))
(if date (format "published on %s\n" date) "\n")
"\n### Notes\n"
)))
(t
(insert (concat
"\n"
(cond
((and url title) (format "** %s (%s)\n" title url))
(url (format "** %s\n" url))
(title (format "** %s\n" title)))
(if authors (format "\n%s" authors))
(if journal (format "\nin %s " journal))
(if date (format "published on %s\n" date) "\n")
"\n*** Notes\n"
)))
)))
(keymap-set consult-web-embark-scholar-actions-map "i n" #'consult-web-embark-scholar-insert-note)Here is an example for opening youtube videos with mpv.io.
(defun my:youtube-mpv (url &rest args)
(interactive (let* ((string (consult-web-dynamic-youtube nil t))
(url (and (stringp string) (get-text-property 0 :url string))))
(list url)))
(if url
(start-process "consult-web-mpv" nil "mpv"
url)))
(defun consult-web-embark-play-with-mpv (cand)
(let* ((url (and (stringp cand) (get-text-property 0 :url cand))))
(my:youtube-mpv url)))
;;;; if you have emacs package for mpv already installed
;; (defun consult-web-embark-play-with-mpv (cand)
;; (let* ((url (and (stringp cand) (get-text-property 0 :url cand))))
;; (mpv-play-url url)))
(keymap-set consult-web-embark-general-actions-map "o m" #'consult-web-embark-play-with-mpv)By default schoalr sources have a different category, but you can redefine the source and introduce new categories with their own embark actions,
Without docstrings and whitespaces the code is less than 1000 lines and it only depends on consult and buil-in url-retrieve.
Although adding other packages (embark, vertico, emacs-request, …) may help the overall workflow, they are not neccessary.
You can only load the parts you need. For example if all you need is an autosuggestion utility, then you can do:
(require 'consult-web-brave-autosuggest)This adds an extra 100-200 lines of code per source.
Here is a table of all the currently implemented sources.
| Source | Category |
|---|---|
| chatGPT | Simple AI prompts |
| Bing | Search Engine |
| Brave | Search Engine |
| Brave AutoSuggest | AutoSuggest |
| consult-line-multi | Local Text in Buffers |
| consult-notes | Local Notes |
| consult-buffer | Local Buffers |
| DuckDuckGo (Limited API) | Search Suggestions |
| Elfeed | Feeds (RSS, videos,…) |
| Search Engine | |
| Google Autosuggest | AutoSuggest |
| gptel | AI Assistant |
| Doi.org | Academic Reference |
| PubMed | Academic Reference |
| Scopus | Academic Reference |
| StackOverflow | Community Forum |
| Wikipedia | Encyclopedia |
| YouTube | Videos |
Lots of customization options. New sources can be added as you wish with different format, different actions,…
Dynamic collection allows for complex workflows on the fly. Change query parameters on the fly by passing arguments. Select a random set of results ad-hoc using embark and run embark actions on them. This allows batch processing as well. For example add a long list of sources for later review.
(defun consult-web-dwim (&optional input &rest args)
(interactive)
(let ((input (or input
(and consult-web-default-autosuggest-command (funcall-interactively consult-web-default-autosuggest-command))
(consult-web--read-search-string))))
(ignore-errors (minibuffer-with-setup-hook
(lambda () (vertico-first) (embark-dwim))
(consult-web-multi input args)))))