Go http client.do returns error and nil response

6 days ago 7
ARTICLE AD BOX

I'm currently working on a web crawler in Go, and it found this URL: https://git.btlr.sh. When checking whether this domain has a robots.txt file, the crawler throws an error.

I've replicated the error in a test script, while just attempting to send a get request to https://git.btlr.sh.

panic: Get "https://git.btlr.sh": EOF goroutine 1 [running]: main.main() /Users/nico/handmade-indexer/robots/test.go:22 +0x100 exit status 2

I'm not too sure what to make of this error, specifically because I can't investigate the response object, because it is nil. The following script replicates this error:

package main import ( "net/http" ) func main() { httpClient := &http.Client{} url := "https://git.btlr.sh" request, err := http.NewRequest("GET", url, nil) if err != nil { panic(err) } resp, err := httpClient.Do(request) if resp == nil { println("RESP IS NIL") } if err != nil { panic(err) } defer resp.Body.Close() }

When the httpClient does the request, the resp object is nil. So I can't investigate the status code of the response, or any further details which may have been provided in the response.

How can I investigate what is going wrong? Why am I receiving an End-of-File error, when it appears there is nothing loading in the first place. Shouldn't that be returning a 404 response? Or since it seems to be pointing to a bash file and not a domain, shouldn't I be receiving a "domain invalid" sort of error?

It's clear to me I'm missing something obvious, but I'm not sure where to look for it.

Read Entire Article