Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic usage only shows a single link, but site is full of them #469

Open
mmmint opened this issue Apr 13, 2020 · 3 comments
Open

Basic usage only shows a single link, but site is full of them #469

mmmint opened this issue Apr 13, 2020 · 3 comments
Labels

Comments

@mmmint
Copy link

mmmint commented Apr 13, 2020

While trying below code I only get a single link, but the website is full of links.

Visiting http://teenage.engineering

What is happening? Thanks for any hints.

package main

import (
	"fmt"
	"github.com/gocolly/colly"
)

func main() {

	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("http://teenage.engineering")
}
@asciimoo
Copy link
Member

The html content is inside a noscript tag and somehow the html parsing lib doesn't handle it. I need further investigation to fix this issue, thanks for reporting.

@asciimoo asciimoo added the bug label Apr 17, 2020
@anthonygedeon
Copy link

anthonygedeon commented Oct 28, 2021

Is this bug still reproducible?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants