19 Commits

Author SHA1 Message Date
Daniel Martí
4cc9890745 add a simple issue template
For now, all it asks for is a few versions, and the typical three
questions to understand a bug.
2019-02-22 00:13:27 +01:00
Daniel Martí
37d13f2933 update all mod dependencies 2019-02-21 23:43:53 +01:00
Daniel Martí
26c9acb5b1 avoid ctx.Done() goroutine leak in Selector.run
As spotted in #162 by a contributor, if the context is done before the
Selector.run caller has received from the channel, the spawned goroutine
may leak if blocked on a send.
2019-02-21 17:58:08 +01:00
Daniel Martí
811d6d54d3 don't run TestFileUpload subtests in parallel
Turns out that these subtests are the only pair which cannot run in
parallel with each other. Undo that change and add a TODO. This should
fix the CI failures.

While at it, remove an unnecessary testAllocate line.
2019-02-21 17:22:40 +01:00
Daniel Martí
4c16288502 skip the error log in TestAllocatePortInUse
We know we're going to see an error, so don't log it too:

	$ go test -run TestAllocatePortInUse
	2019/02/21 17:11:34 ERROR: pool could not allocate runner ...
	PASS
	ok      github.com/chromedp/chromedp    0.004s
2019-02-21 17:13:38 +01:00
Daniel Martí
da4f783362 make all tests run in parallel
The subtests were almost all marked as parallel, but that's not enough.
That only makes the subtests run in parallel with other subtests within
the same tests, not with any other test.

Since none of the tests make use of globals nor require the entire
program to themselves, properly run all the tests in parallel.

Speeds up 'go test' on my 8-core laptop from an average of ~130s to an
average of ~50s. Many tests hit timeouts and have sleeps, so we want to
avoid running those sequentially whenever possible.
2019-02-21 13:56:54 +01:00
Daniel Martí
5ca52f3e1b use runner.LookChromeNames in TestMain
It supports alternative names for Chrome such as chromium, as well as
extra names to look for like headless-shell.

Also swap the os.Getenv logic, so that we only do the exec.LookPath work
if the env var is unset.
2019-02-21 13:53:05 +01:00
zhongjiajia
5dc1e0f3af use buffered chan for h.detached 2019-02-20 13:12:41 +01:00
Bob Potter
85ecf4f31f chromedp: fix SetHandlerByID
Don't fall through and return an error if we found a handler with a
matching ID.
2019-02-20 13:01:21 +01:00
Daniel Martí
7f54f3f93c CI: test on 1.11.x instead of tip
tip is rather unstable, so we shouldn't block PRs if it happens to break
our build or tests.

While at it, run 'go mod tidy' with the latest tip version.
2019-01-14 10:38:19 +00:00
Daniel Martí
98d4b0de6e pool: error quickly if we find a port in use
Before the fix, the added test would give a Pool.Allocate error like:

	pool could not connect to 9000: timeout waiting for initial target

The actual underlying error, which can only be seen if one inspects
chrome's stderr, is that it failed to bind to the debugging protocol
port if it was already in use.

This is of course an issue with the environment that chromedp is being
run under, since it was given a port range that wasn't available.
However, the confusing error can lead to developers wasting their time
instead of spotting the error quickly.

Unfortunately, there doesn't seem to be a way to have Chrome exit
immediately if it can't bind to the given port. So, instead of relying
on it, check if the current process can bind to the port first.

Add a test too, where we grab the first port in the pool range, and
check that we get an error that's not confusing.

Fixes #253.
2018-12-01 11:54:16 +00:00
Kenneth Shaw
bf52fed0d3 Fixing windows build issue 2018-07-18 06:19:22 +07:00
Kenneth Shaw
34591780d9 Updating dependencies 2018-07-13 12:49:42 +07:00
Kenneth Shaw
53015e7d81 Changing Debugging => DevTools 2018-07-13 12:46:30 +07:00
Kenneth Shaw
74e379587b Minor code cleanup to client package before rewrite 2018-07-13 11:35:03 +07:00
Kenneth Shaw
4ae10864e4 Adding client API changes prior to package rewrite 2018-07-13 11:24:37 +07:00
Kenneth Shaw
d413f67302 Minor cleanup to client/gen.go 2018-07-13 11:00:35 +07:00
Kenneth Shaw
622c90c82c Cleaning up runner API prior to package rewrite 2018-07-13 10:57:20 +07:00
Kenneth Shaw
e051c4a982 Removing errors dependency in client and runner packages 2018-07-13 09:28:45 +07:00
23 changed files with 339 additions and 340 deletions

15
.github/ISSUE_TEMPLATE vendored Normal file
View File

@@ -0,0 +1,15 @@
#### What versions are you running?
<pre>
$ go list -m github.com/chromedp/chromedp
$ chromium --version
$ go version
</pre>
#### What did you do?
#### What did you expect to see?
#### What did you see instead?

View File

@@ -1,7 +1,7 @@
language: go
go:
- 1.10.x
- tip
- 1.11.x
addons:
apt:
chrome: stable

View File

@@ -1,102 +1,18 @@
# About chromedp [![Build Status][1]][2] [![Coverage Status][3]][4]
Package chromedp is a faster, simpler way to drive browsers in Go using the
[Chrome Debugging Protocol][5] (for Chrome, Edge, Safari, etc) without external
dependencies (ie, Selenium, PhantomJS, etc).
**NOTE:** chromedp's API is currently unstable, and may change at a moments
notice. There are likely extremely bad bugs lurking in this code. **CAVEAT USER**.
Package chromedp is a faster, simpler way to drive browsers supporting the
[Chrome DevTools Protocol][5] in Go using the without external dependencies
(ie, Selenium, PhantomJS, etc).
## Installing
Install in the usual way:
Install in the usual Go way:
```sh
go get -u github.com/chromedp/chromedp
```
## Using
Below is a simple Google search performed using chromedp (taken from
[examples/simple][6]):
This example shows logic for a simple search for a known website, clicking on
the right link, and then taking a screenshot of a specific element on the
loaded page and saving that to a local file on disk.
```go
// Command simple is a chromedp example demonstrating how to do a simple google
// search.
package main
import (
"context"
"fmt"
"io/ioutil"
"log"
"time"
"github.com/chromedp/cdproto/cdp"
"github.com/chromedp/chromedp"
)
func main() {
var err error
// create context
ctxt, cancel := context.WithCancel(context.Background())
defer cancel()
// create chrome instance
c, err := chromedp.New(ctxt, chromedp.WithLog(log.Printf))
if err != nil {
log.Fatal(err)
}
// run task list
var site, res string
err = c.Run(ctxt, googleSearch("site:brank.as", "Home", &site, &res))
if err != nil {
log.Fatal(err)
}
// shutdown chrome
err = c.Shutdown(ctxt)
if err != nil {
log.Fatal(err)
}
// wait for chrome to finish
err = c.Wait()
if err != nil {
log.Fatal(err)
}
log.Printf("saved screenshot from search result listing `%s` (%s)", res, site)
}
func googleSearch(q, text string, site, res *string) chromedp.Tasks {
var buf []byte
sel := fmt.Sprintf(`//a[text()[contains(., '%s')]]`, text)
return chromedp.Tasks{
chromedp.Navigate(`https://www.google.com`),
chromedp.WaitVisible(`#hplogo`, chromedp.ByID),
chromedp.SendKeys(`#lst-ib`, q+"\n", chromedp.ByID),
chromedp.WaitVisible(`#res`, chromedp.ByID),
chromedp.Text(sel, res),
chromedp.Click(sel),
chromedp.WaitNotVisible(`.preloader-content`, chromedp.ByQuery),
chromedp.WaitVisible(`a[href*="twitter"]`, chromedp.ByQuery),
chromedp.Location(site),
chromedp.ScrollIntoView(`.banner-section.third-section`, chromedp.ByQuery),
chromedp.Sleep(2 * time.Second), // wait for animation to finish
chromedp.Screenshot(`.banner-section.third-section`, &buf, chromedp.ByQuery),
chromedp.ActionFunc(func(context.Context, cdp.Executor) error {
return ioutil.WriteFile("screenshot.png", buf, 0644)
}),
}
}
```
## Examples
Please see the [examples][6] project for more examples. Please refer to the
[GoDoc API listing][7] for a summary of the API and Actions.
@@ -104,11 +20,11 @@ Please see the [examples][6] project for more examples. Please refer to the
## Resources
* [chromedp: A New Way to Drive the Web][8] - GopherCon SG 2017 talk
* [Chrome DevTools Protocol][5] - Chrome Debugging Protocol Domain documentation
* [Chrome DevTools Protocol][5] - Chrome DevTools Protocol Domain documentation
* [chromedp examples][6] - various `chromedp` examples
* [`github.com/chromedp/cdproto`][9] - GoDoc listing for the CDP domains used by `chromedp`
* [`github.com/chromedp/cdproto-gen`][10] - tool used to generate `cdproto`
* [`github.com/chromedp/chromedp-proxy`][11] - a simple CDP proxy for logging/debugging CDP clients and browser instances
* [`github.com/chromedp/chromedp-proxy`][11] - a simple CDP proxy for logging CDP clients and browsers
## TODO

View File

@@ -1,9 +1,9 @@
// Package chromedp is a high level Chrome Debugging Protocol domain manager
// that simplifies driving web browsers (Chrome, Safari, Edge, Android Web
// Views, and others) for scraping, unit testing, or profiling web pages.
// Package chromedp is a high level Chrome DevTools Protocol client that
// simplifies driving browsers for scraping, unit testing, or profiling web
// pages using the CDP.
//
// chromedp requires no third-party dependencies (ie, Selenium), implementing
// the async Chrome Debugging Protocol natively.
// chromedp requires no third-party dependencies, implementing the async Chrome
// DevTools Protocol entirely in Go.
package chromedp
import (
@@ -35,8 +35,9 @@ const (
DefaultPoolEndPort = 10000
)
// CDP contains information for managing a Chrome process runner, low level
// JSON and websocket client, and associated network, page, and DOM handling.
// CDP is the high-level Chrome DevTools Protocol browser manager, handling the
// browser process runner, WebSocket clients, associated targets, and network,
// page, and DOM events.
type CDP struct {
// r is the chrome runner.
r *runner.Runner
@@ -90,7 +91,7 @@ func New(ctxt context.Context, opts ...Option) (*CDP, error) {
// watch handlers
if c.watch == nil {
c.watch = c.r.WatchPageTargets(ctxt)
c.watch = c.r.Client().WatchPageTargets(ctxt)
}
go func() {
@@ -238,6 +239,7 @@ func (c *CDP) SetHandlerByID(id string) error {
if i, ok := c.handlerMap[id]; ok {
c.cur = c.handlers[i]
return nil
}
return fmt.Errorf("no handler associated with target id %s", id)
@@ -339,7 +341,7 @@ func (c *CDP) Run(ctxt context.Context, a Action) error {
return a.Do(ctxt, cur)
}
// Option is a Chrome Debugging Protocol option.
// Option is a Chrome DevTools Protocol option.
type Option func(*CDP) error
// WithRunner is a CDP option to specify the underlying Chrome runner to

View File

@@ -4,7 +4,6 @@ import (
"context"
"log"
"os"
"os/exec"
"path"
"testing"
"time"
@@ -78,16 +77,9 @@ func TestMain(m *testing.M) {
// its worth noting that newer versions of chrome (64+) run much faster
// than older ones -- same for headless_shell ...
execPath := runner.DefaultChromePath
if testRunner := os.Getenv("CHROMEDP_TEST_RUNNER"); testRunner != "" {
execPath = testRunner
} else {
// use headless_shell, if on path
var hsPath string
hsPath, err = exec.LookPath("headless_shell")
if err == nil {
execPath = hsPath
}
execPath := os.Getenv("CHROMEDP_TEST_RUNNER")
if execPath == "" {
execPath = runner.LookChromeNames("headless_shell")
}
cliOpts = append(cliOpts, runner.ExecPath(execPath))

View File

@@ -4,7 +4,7 @@ import "fmt"
//go:generate easyjson -omit_empty -output_filename easyjson.go chrome.go
// Chrome holds connection information for a Chrome, Edge, or Safari target.
// Chrome holds connection information for a Chrome target.
//
//easyjson:json
type Chrome struct {
@@ -20,7 +20,7 @@ type Chrome struct {
// String satisfies the stringer interface.
func (c Chrome) String() string {
return fmt.Sprintf("%s (`%s`)", c.ID, c.Title)
return fmt.Sprintf("[%s]: %q", c.ID, c.Title)
}
// GetID returns the target ID.
@@ -33,6 +33,12 @@ func (c *Chrome) GetType() TargetType {
return c.Type
}
// GetDevtoolsURL returns the devtools frontend target URL, satisfying the
// domains.Target interface.
func (c *Chrome) GetDevtoolsURL() string {
return c.DevtoolsURL
}
// GetWebsocketURL provides the websocket URL for the target, satisfying the
// domains.Target interface.
func (c *Chrome) GetWebsocketURL() string {

View File

@@ -1,5 +1,4 @@
// Package client provides the low level Chrome Debugging Protocol JSON types
// and related funcs.
// Package client provides the low level Chrome DevTools Protocol client.
package client
//go:generate go run gen.go
@@ -45,15 +44,16 @@ const (
ErrUnsupportedProtocolVersion Error = "unsupported protocol version"
)
// Target is the common interface for a Chrome Debugging Protocol target.
// Target is the common interface for a Chrome DevTools Protocol target.
type Target interface {
String() string
GetID() string
GetType() TargetType
GetDevtoolsURL() string
GetWebsocketURL() string
}
// Client is a Chrome Debugging Protocol client.
// Client is a Chrome DevTools Protocol client.
type Client struct {
url string
check time.Duration
@@ -63,7 +63,7 @@ type Client struct {
rw sync.RWMutex
}
// New creates a new Chrome Debugging Protocol client.
// New creates a new Chrome DevTools Protocol client.
func New(opts ...Option) *Client {
c := &Client{
url: DefaultEndpoint,
@@ -120,8 +120,7 @@ func (c *Client) ListTargets(ctxt context.Context) ([]Target, error) {
var err error
var l []json.RawMessage
err = c.doReq(ctxt, "list", &l)
if err != nil {
if err = c.doReq(ctxt, "list", &l); err != nil {
return nil, err
}
@@ -200,8 +199,7 @@ func (c *Client) newTarget(ctxt context.Context, buf []byte) (Target, error) {
case "chrome", "chromium", "microsoft edge", "safari", "":
x := new(Chrome)
if buf != nil {
err = easyjson.Unmarshal(buf, x)
if err != nil {
if err = easyjson.Unmarshal(buf, x); err != nil {
return nil, err
}
}
@@ -226,8 +224,7 @@ func (c *Client) NewPageTargetWithURL(ctxt context.Context, urlstr string) (Targ
u += "?" + urlstr
}
err = c.doReq(ctxt, u, t)
if err != nil {
if err = c.doReq(ctxt, u, t); err != nil {
return nil, err
}
@@ -251,23 +248,15 @@ func (c *Client) CloseTarget(ctxt context.Context, t Target) error {
// VersionInfo returns information about the remote debugging protocol.
func (c *Client) VersionInfo(ctxt context.Context) (map[string]string, error) {
var err error
v := map[string]string{}
err = c.doReq(ctxt, "version", &v)
if err != nil {
v := make(map[string]string)
if err := c.doReq(ctxt, "version", &v); err != nil {
return nil, err
}
return v, nil
}
// WatchPageTargets watches for new page targets.
func (c *Client) WatchPageTargets(ctxt context.Context) <-chan Target {
if ctxt == nil {
ctxt = context.Background()
}
ch := make(chan Target)
go func() {
defer close(ch)
@@ -311,10 +300,11 @@ func (c *Client) WatchPageTargets(ctxt context.Context) <-chan Target {
return ch
}
// Option is a Chrome Debugging Protocol client option.
// Option is a Chrome DevTools Protocol client option.
type Option func(*Client)
// URL is a client option to specify the remote Chrome instance to connect to.
// URL is a client option to specify the remote Chrome DevTools Protocol
// instance to connect to.
func URL(urlstr string) Option {
return func(c *Client) {
// since chrome 66+, dev tools requires the host name to be either an

View File

@@ -26,17 +26,26 @@ const (
var (
flagOut = flag.String("out", "targettype.go", "out file")
typeAsStringRE = regexp.MustCompile(`type_as_string\s+==\s+"([^"]+)"`)
)
func main() {
flag.Parse()
if err := run(); err != nil {
log.Fatal(err)
}
}
var typeAsStringRE = regexp.MustCompile(`type_as_string\s+==\s+"([^"]+)"`)
// run executes the generator.
func run() error {
var err error
// grab source
buf, err := grab(devtoolsHTTPClientCc)
if err != nil {
log.Fatal(err)
return err
}
// find names
@@ -55,15 +64,11 @@ func main() {
decodeVals += fmt.Sprintf("case %s:\n*tt=%s\n", name, name)
}
err = ioutil.WriteFile(*flagOut, []byte(fmt.Sprintf(targetTypeSrc, constVals, decodeVals)), 0644)
if err != nil {
log.Fatal(err)
if err = ioutil.WriteFile(*flagOut, []byte(fmt.Sprintf(targetTypeSrc, constVals, decodeVals)), 0644); err != nil {
return err
}
err = exec.Command("gofmt", "-w", "-s", *flagOut).Run()
if err != nil {
log.Fatal(err)
}
return exec.Command("gofmt", "-w", "-s", *flagOut).Run()
}
// grab retrieves a file from the chromium source code.
@@ -93,7 +98,6 @@ const (
// Code generated by gen.go. DO NOT EDIT.
import (
// "errors"
easyjson "github.com/mailru/easyjson"
jlexer "github.com/mailru/easyjson/jlexer"
jwriter "github.com/mailru/easyjson/jwriter"
@@ -129,7 +133,6 @@ func (tt *TargetType) UnmarshalEasyJSON(in *jlexer.Lexer) {
%s
default:
// in.AddError(errors.New("unknown TargetType"))
*tt = z
}
}

View File

@@ -3,7 +3,6 @@ package client
// Code generated by gen.go. DO NOT EDIT.
import (
// "errors"
easyjson "github.com/mailru/easyjson"
jlexer "github.com/mailru/easyjson/jlexer"
jwriter "github.com/mailru/easyjson/jwriter"
@@ -70,7 +69,6 @@ func (tt *TargetType) UnmarshalEasyJSON(in *jlexer.Lexer) {
*tt = Worker
default:
// in.AddError(errors.New("unknown TargetType"))
*tt = z
}
}

View File

@@ -6,7 +6,7 @@ import (
"github.com/gorilla/websocket"
)
const (
var (
// DefaultReadBufferSize is the default maximum read buffer size.
DefaultReadBufferSize = 25 * 1024 * 1024
@@ -14,7 +14,7 @@ const (
DefaultWriteBufferSize = 10 * 1024 * 1024
)
// Transport is the common interface to send/receive messages.
// Transport is the common interface to send/receive messages to a target.
type Transport interface {
Read() ([]byte, error)
Write([]byte) error
@@ -43,7 +43,7 @@ func (c *Conn) Write(buf []byte) error {
// Dial dials the specified target's websocket URL.
//
// Note: uses gorilla/websocket.
func Dial(t Target, opts ...DialOption) (Transport, error) {
func Dial(urlstr string, opts ...DialOption) (Transport, error) {
d := &websocket.Dialer{
ReadBufferSize: DefaultReadBufferSize,
WriteBufferSize: DefaultWriteBufferSize,
@@ -55,7 +55,7 @@ func Dial(t Target, opts ...DialOption) (Transport, error) {
}
// connect
conn, _, err := d.Dial(t.GetWebsocketURL(), nil)
conn, _, err := d.Dial(urlstr, nil)
if err != nil {
return nil, err
}
@@ -65,5 +65,3 @@ func Dial(t Target, opts ...DialOption) (Transport, error) {
// DialOption is a dial option.
type DialOption func(*websocket.Dialer)
// TODO: add dial options ...

11
go.mod
View File

@@ -1,10 +1,9 @@
module github.com/chromedp/chromedp
require (
github.com/chromedp/cdproto v0.0.0-20180703215205-c125a34ea3b3
github.com/disintegration/imaging v1.4.2
github.com/gorilla/websocket v1.2.0
github.com/knq/sysutil v0.0.0-20180306023629-0218e141a794
github.com/mailru/easyjson v0.0.0-20180606163543-3fdea8d05856
golang.org/x/image v0.0.0-20180628062038-cc896f830ced
github.com/chromedp/cdproto v0.0.0-20190217000753-2d8e8962ceb2
github.com/disintegration/imaging v1.6.0
github.com/gorilla/websocket v1.4.0
github.com/mailru/easyjson v0.0.0-20190221075403-6243d8e04c3f
golang.org/x/image v0.0.0-20190220214146-31aff87c08e9 // indirect
)

25
go.sum
View File

@@ -1,9 +1,16 @@
github.com/chromedp/cdproto v0.0.0-20180522032958-55db67b53f25/go.mod h1:C2GPAraqdt1KfZU7aSmx1XUgarNq/3JmxevQkmCjOVs=
github.com/chromedp/cdproto v0.0.0-20180703215205-c125a34ea3b3/go.mod h1:C2GPAraqdt1KfZU7aSmx1XUgarNq/3JmxevQkmCjOVs=
github.com/disintegration/imaging v1.4.2/go.mod h1:9B/deIUIrliYkyMTuXJd6OUFLcrZ2tf+3Qlwnaf/CjU=
github.com/gorilla/websocket v1.2.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/knq/sysutil v0.0.0-20180306023629-0218e141a794/go.mod h1:BjPj+aVjl9FW/cCGiF3nGh5v+9Gd3VCgBQbod/GlMaQ=
github.com/mailru/easyjson v0.0.0-20180323154445-8b799c424f57/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.0.0-20180606163543-3fdea8d05856/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
golang.org/x/image v0.0.0-20180403161127-f315e4403028/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
golang.org/x/image v0.0.0-20180628062038-cc896f830ced/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
github.com/chromedp/cdproto v0.0.0-20190217000753-2d8e8962ceb2 h1:4Ck8YOuS0G3+0xMb80cDSff7QpUolhSc0PGyfagbcdA=
github.com/chromedp/cdproto v0.0.0-20190217000753-2d8e8962ceb2/go.mod h1:xquOK9dIGFlLaIGI4c6IyfLI/Gz0LiYYuJtzhsUODgI=
github.com/disintegration/imaging v1.6.0 h1:nVPXRUUQ36Z7MNf0O77UzgnOb1mkMMor7lmJMJXc/mA=
github.com/disintegration/imaging v1.6.0/go.mod h1:xuIt+sRxDFrHS0drzXUlCJthkJ8k7lkkUojDSR247MQ=
github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q=
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/knq/sysutil v0.0.0-20181215143952-f05b59f0f307 h1:vl4eIlySbjertFaNwiMjXsGrFVK25aOWLq7n+3gh2ls=
github.com/knq/sysutil v0.0.0-20181215143952-f05b59f0f307/go.mod h1:BjPj+aVjl9FW/cCGiF3nGh5v+9Gd3VCgBQbod/GlMaQ=
github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.0.0-20190221075403-6243d8e04c3f h1:B6PQkurxGG1rqEX96oE14gbj8bqvYC5dtks9r5uGmlE=
github.com/mailru/easyjson v0.0.0-20190221075403-6243d8e04c3f/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81 h1:00VmoueYNlNz/aHIilyyQz/MHSqGoWJzpFv/HW8xpzI=
golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
golang.org/x/image v0.0.0-20190220214146-31aff87c08e9 h1:+vH8qNweCrORN49012OX3h0oWEXO3p+rRnpAGQinddk=
golang.org/x/image v0.0.0-20190220214146-31aff87c08e9/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=

View File

@@ -24,7 +24,7 @@ import (
"github.com/chromedp/chromedp/client"
)
// TargetHandler manages a Chrome Debugging Protocol target.
// TargetHandler manages a Chrome DevTools Protocol target.
type TargetHandler struct {
conn client.Transport
@@ -64,7 +64,7 @@ type TargetHandler struct {
// NewTargetHandler creates a new handler for the specified client target.
func NewTargetHandler(t client.Target, logf, debugf, errf func(string, ...interface{})) (*TargetHandler, error) {
conn, err := client.Dial(t)
conn, err := client.Dial(t.GetWebsocketURL())
if err != nil {
return nil, err
}
@@ -89,7 +89,7 @@ func (h *TargetHandler) Run(ctxt context.Context) error {
h.qres = make(chan *cdproto.Message)
h.qevents = make(chan *cdproto.Message)
h.res = make(map[int64]chan *cdproto.Message)
h.detached = make(chan *inspector.EventDetached)
h.detached = make(chan *inspector.EventDetached, 1)
h.pageWaitGroup = new(sync.WaitGroup)
h.domWaitGroup = new(sync.WaitGroup)
h.Unlock()

View File

@@ -21,10 +21,10 @@ const (
)
func TestMouseClickXY(t *testing.T) {
var err error
t.Parallel()
var err error
c := testAllocate(t, "input.html")
defer c.Release()
@@ -78,6 +78,8 @@ func TestMouseClickXY(t *testing.T) {
}
func TestMouseClickNode(t *testing.T) {
t.Parallel()
tests := []struct {
sel, exp string
opt MouseOption
@@ -128,6 +130,8 @@ func TestMouseClickNode(t *testing.T) {
}
func TestMouseClickOffscreenNode(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
exp int
@@ -186,6 +190,8 @@ func TestMouseClickOffscreenNode(t *testing.T) {
}
func TestKeyAction(t *testing.T) {
t.Parallel()
tests := []struct {
sel, exp string
by QueryOption
@@ -238,6 +244,8 @@ func TestKeyAction(t *testing.T) {
}
func TestKeyActionNode(t *testing.T) {
t.Parallel()
tests := []struct {
sel, exp string
by QueryOption

19
pool.go
View File

@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"log"
"net"
"sync"
"github.com/chromedp/chromedp/runner"
@@ -70,11 +71,27 @@ func (p *Pool) Allocate(ctxt context.Context, opts ...runner.CommandLineOption)
r := p.next(ctxt)
// Check if the port is available first. If it's not, Chrome will print
// an "address already in use" error, but it will otherwise keep
// running. This can lead to Allocate succeeding, while the chrome
// process isn't actually listening on the port we need.
l, err := net.Listen("tcp", fmt.Sprintf("localhost:%d", r.port))
if err != nil {
// we can't use this port, e.g. address already in use
p.errf("pool could not allocate runner on port %d: %v", r.port, err)
return nil, err
}
l.Close()
p.debugf("pool allocating %d", r.port)
// create runner
r.r, err = runner.New(append([]runner.CommandLineOption{
runner.HeadlessPathPort("", r.port),
runner.ExecPath(runner.LookChromeNames("headless_shell")),
runner.RemoteDebuggingPort(r.port),
runner.NoDefaultBrowserCheck,
runner.NoFirstRun,
runner.Headless,
}, opts...)...)
if err != nil {
defer r.Release()

47
pool_test.go Normal file
View File

@@ -0,0 +1,47 @@
package chromedp
import (
"context"
"net"
"strconv"
"strings"
"testing"
)
func TestAllocatePortInUse(t *testing.T) {
t.Parallel()
// take a random available port
l, err := net.Listen("tcp4", "localhost:0")
if err != nil {
t.Fatal(err)
}
defer l.Close()
ctxt, cancel := context.WithCancel(context.Background())
defer cancel()
// make the pool use the port already in use via a port range
_, portStr, _ := net.SplitHostPort(l.Addr().String())
port, _ := strconv.Atoi(portStr)
pool, err := NewPool(
PortRange(port, port+1),
// skip the error log from the used port
PoolLog(nil, nil, func(string, ...interface{}) {}),
)
if err != nil {
t.Fatal(err)
}
c, err := pool.Allocate(ctxt)
if err != nil {
want := "address already in use"
got := err.Error()
if !strings.Contains(got, want) {
t.Fatalf("wanted error to contain %q, but got %q", want, got)
}
} else {
t.Fatal("wanted Allocate to error if port is in use")
c.Release()
}
}

View File

@@ -190,6 +190,8 @@ func TestText(t *testing.T) {
}
func TestClear(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -244,6 +246,8 @@ func TestClear(t *testing.T) {
}
func TestReset(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -315,6 +319,8 @@ func TestValue(t *testing.T) {
}
func TestSetValue(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -442,6 +448,8 @@ func TestAttributesAll(t *testing.T) {
}
func TestSetAttributes(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -547,6 +555,8 @@ func TestAttributeValue(t *testing.T) {
}
func TestSetAttributeValue(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -589,6 +599,8 @@ func TestSetAttributeValue(t *testing.T) {
}
func TestRemoveAttribute(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -626,6 +638,8 @@ func TestRemoveAttribute(t *testing.T) {
}
func TestClick(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -666,6 +680,8 @@ func TestClick(t *testing.T) {
}
func TestDoubleClick(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -703,6 +719,8 @@ func TestDoubleClick(t *testing.T) {
}
func TestSendKeys(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -773,6 +791,8 @@ func TestScreenshot(t *testing.T) {
}
func TestSubmit(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -813,6 +833,8 @@ func TestSubmit(t *testing.T) {
}
func TestComputedStyle(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -870,6 +892,8 @@ func TestComputedStyle(t *testing.T) {
}
func TestMatchedStyle(t *testing.T) {
t.Parallel()
tests := []struct {
sel string
by QueryOption
@@ -940,9 +964,6 @@ func TestFileUpload(t *testing.T) {
t.Fatal(err)
}
c := testAllocate(t, "")
defer c.Release()
tests := []struct {
a Action
}{
@@ -952,6 +973,10 @@ func TestFileUpload(t *testing.T) {
for i, test := range tests {
t.Run(fmt.Sprintf("test %d", i), func(t *testing.T) {
// TODO: refactor the test so the subtests can run in
// parallel
//t.Parallel()
c := testAllocate(t, "")
defer c.Release()

View File

@@ -3,10 +3,11 @@
package runner
const (
// DefaultChromePath is the default path to the Chrome application.
// DefaultChromePath is the default path to use for Chrome if the
// executable is not in $PATH.
DefaultChromePath = `/Applications/Google Chrome.app/Contents/MacOS/Google Chrome`
)
func findChromePath() string {
return DefaultChromePath
}
// DefaultChromeNames are the default Chrome executable names to look for in
// $PATH.
var DefaultChromeNames []string

View File

@@ -2,30 +2,18 @@
package runner
import "os/exec"
const (
// DefaultChromePath is the default path to the google-chrome executable if
// a variant cannot be found on $PATH.
// DefaultChromePath is the default path to use for Chrome if the
// executable is not in $PATH.
DefaultChromePath = "/usr/bin/google-chrome"
)
// chromeNames are the Chrome executable names to search for in the path.
var chromeNames = []string{
// DefaultChromeNames are the default Chrome executable names to look for in
// $PATH.
var DefaultChromeNames = []string{
"google-chrome",
"chromium-browser",
"chromium",
"google-chrome-beta",
"google-chrome-unstable",
}
func findChromePath() string {
for _, p := range chromeNames {
path, err := exec.LookPath(p)
if err == nil {
return path
}
}
return DefaultChromePath
}

View File

@@ -2,32 +2,12 @@
package runner
import "os/exec"
const (
// DefaultChromePath is the default path to use for Google Chrome if the
// DefaultChromePath is the default path to use for Chrome if the
// executable is not in %PATH%.
DefaultChromePath = `C:\Program Files (x86)\Google\Chrome\Application\chrome.exe`
// DefaultEdgeDiagnosticsAdapterPath is the default path to use for the
// Microsoft Edge Diagnostics Adapter if the executable is not in %PATH%.
DefaultEdgeDiagnosticsAdapterPath = `c:\Edge\EdgeDiagnosticsAdapter\x64\EdgeDiagnosticsAdapter.exe`
)
func findChromePath() string {
path, err := exec.LookPath(`chrome.exe`)
if err == nil {
return path
}
return DefaultChromePath
}
func findEdgePath() string {
path, err := exec.LookPath(`EdgeDiagnosticsAdapter.exe`)
if err == nil {
return path
}
return DefaultEdgeDiagnosticsAdapterPath
}
// DefaultChromeNames are the default Chrome executable names to look for in
// %PATH%.
var DefaultChromeNames = []string{`chrome.exe`}

View File

@@ -3,10 +3,8 @@ package runner
import (
"context"
"errors"
"fmt"
"io/ioutil"
"net/url"
"os"
"os/exec"
"regexp"
@@ -22,6 +20,35 @@ const (
DefaultUserDataDirPrefix = "chromedp-runner.%d."
)
// Error is a runner error.
type Error string
// Error satisfies the error interface.
func (err Error) Error() string {
return string(err)
}
// Error values.
const (
// ErrAlreadyStarted is the already started error.
ErrAlreadyStarted Error = "already started"
// ErrAlreadyWaiting is the already waiting error.
ErrAlreadyWaiting Error = "already waiting"
// ErrInvalidURLs is the invalid url-opts error.
ErrInvalidURLOpts Error = "invalid url-opts"
// ErrInvalidCmdOpts is the invalid cmd-opts error.
ErrInvalidCmdOpts Error = "invalid cmd-opts"
// ErrInvalidProcessOpts is the invalid process-opts error.
ErrInvalidProcessOpts Error = "invalid process-opts"
// ErrInvalidExecPath is the invalid exec-path error.
ErrInvalidExecPath Error = "invalid exec-path"
)
// Runner holds information about a running Chrome process.
type Runner struct {
opts map[string]interface{}
@@ -34,19 +61,18 @@ type Runner struct {
func New(opts ...CommandLineOption) (*Runner, error) {
var err error
cliOpts := map[string]interface{}{}
cliOpts := make(map[string]interface{})
// apply opts
for _, o := range opts {
err = o(cliOpts)
if err != nil {
if err = o(cliOpts); err != nil {
return nil, err
}
}
// set default Chrome options if exec-path not provided
if _, ok := cliOpts["exec-path"]; !ok {
cliOpts["exec-path"] = findChromePath()
cliOpts["exec-path"] = LookChromeNames()
for k, v := range map[string]interface{}{
"no-first-run": true,
"no-default-browser-check": true,
@@ -61,8 +87,7 @@ func New(opts ...CommandLineOption) (*Runner, error) {
// add KillProcessGroup and ForceKill if no other cmd opts provided
if _, ok := cliOpts["cmd-opts"]; !ok {
for _, o := range []CommandLineOption{KillProcessGroup, ForceKill} {
err = o(cliOpts)
if err != nil {
if err = o(cliOpts); err != nil {
return nil, err
}
}
@@ -79,9 +104,9 @@ var cliOptRE = regexp.MustCompile(`^[a-z0-9\-]+$`)
// buildOpts generates the command line options for Chrome.
func (r *Runner) buildOpts() []string {
var opts []string
var urls []string
// process options
var urlstr string
// process opts
for k, v := range r.opts {
if !cliOptRE.MatchString(k) || v == nil {
continue
@@ -91,8 +116,8 @@ func (r *Runner) buildOpts() []string {
case "exec-path", "cmd-opts", "process-opts":
continue
case "start-url":
urlstr = v.(string)
case "url-opts":
urls = v.([]string)
default:
switch z := v.(type) {
@@ -110,16 +135,16 @@ func (r *Runner) buildOpts() []string {
}
}
if urlstr == "" {
urlstr = "about:blank"
if urls == nil {
urls = append(urls, "about:blank")
}
return append(opts, urlstr)
return append(opts, urls...)
}
// Start starts a Chrome process using the specified context. The Chrome
// process can be terminated by closing the passed context.
func (r *Runner) Start(ctxt context.Context) error {
func (r *Runner) Start(ctxt context.Context, opts ...string) error {
var err error
var ok bool
@@ -128,12 +153,7 @@ func (r *Runner) Start(ctxt context.Context) error {
r.rw.RUnlock()
if cmd != nil {
return errors.New("already started")
}
// setup context
if ctxt == nil {
ctxt = context.Background()
return ErrAlreadyStarted
}
// set user data dir, if not provided
@@ -147,36 +167,41 @@ func (r *Runner) Start(ctxt context.Context) error {
}
}
// ensure exec-path set
execPath, ok := r.opts["exec-path"]
// get exec path
var execPath string
if p, ok := r.opts["exec-path"]; ok {
execPath, ok = p.(string)
if !ok {
return errors.New("exec-path command line option not set, or chrome executable not found in $PATH")
return ErrInvalidExecPath
}
}
// ensure execPath is valid
if execPath == "" {
return ErrInvalidExecPath
}
// create cmd
r.cmd = exec.CommandContext(ctxt, execPath.(string), r.buildOpts()...)
r.cmd = exec.CommandContext(ctxt, execPath, append(r.buildOpts(), opts...)...)
// apply cmd opts
if cmdOpts, ok := r.opts["cmd-opts"]; ok {
for _, co := range cmdOpts.([]func(*exec.Cmd) error) {
err = co(r.cmd)
if err != nil {
if err = co(r.cmd); err != nil {
return err
}
}
}
// start process
err = r.cmd.Start()
if err != nil {
if err = r.cmd.Start(); err != nil {
return err
}
// apply process opts
if processOpts, ok := r.opts["process-opts"]; ok {
for _, po := range processOpts.([]func(*os.Process) error) {
err = po(r.cmd.Process)
if err != nil {
if err = po(r.cmd.Process); err != nil {
// TODO: do something better here, as we want to kill
// the child process, do cleanup, etc.
panic(err)
@@ -235,7 +260,7 @@ func (r *Runner) Wait() error {
r.rw.RUnlock()
if waiting {
return errors.New("already waiting")
return ErrAlreadyWaiting
}
r.rw.Lock()
@@ -272,7 +297,7 @@ func (r *Runner) Port() int {
return p
}
// Client returns a Chrome Debugging Protocol client for the running Chrome
// Client returns a Chrome DevTools Protocol client for the running Chrome
// process.
func (r *Runner) Client(opts ...client.Option) *client.Client {
return client.New(append(opts,
@@ -280,12 +305,6 @@ func (r *Runner) Client(opts ...client.Option) *client.Client {
)...)
}
// WatchPageTargets returns a channel that will receive new page targets as
// they are created.
func (r *Runner) WatchPageTargets(ctxt context.Context, opts ...client.Option) <-chan client.Target {
return r.Client(opts...).WatchPageTargets(ctxt)
}
// Run starts a new Chrome process runner, using the provided context and
// command line options.
func Run(ctxt context.Context, opts ...CommandLineOption) (*Runner, error) {
@@ -298,20 +317,19 @@ func Run(ctxt context.Context, opts ...CommandLineOption) (*Runner, error) {
}
// start
err = r.Start(ctxt)
if err != nil {
if err = r.Start(ctxt); err != nil {
return nil, err
}
return r, nil
}
// CommandLineOption is a Chrome command line option.
// CommandLineOption is a runner command line option.
//
// see: http://peter.sh/experiments/chromium-command-line-switches/
type CommandLineOption func(map[string]interface{}) error
// Flag is a generic Chrome command line option to pass a name=value flag to
// Flag is a generic command line option to pass a name=value flag to
// Chrome.
func Flag(name string, value interface{}) CommandLineOption {
return func(m map[string]interface{}) error {
@@ -335,33 +353,12 @@ func Path(path string) CommandLineOption {
}
}
// HeadlessPathPort is the Chrome command line option to set the default
// settings for running the headless_shell executable. If path is empty, then
// an attempt will be made to find headless_shell on the path.
func HeadlessPathPort(path string, port int) CommandLineOption {
if path == "" {
path, _ = exec.LookPath("headless_shell")
}
return func(m map[string]interface{}) error {
m["exec-path"] = path
m["remote-debugging-port"] = port
m["headless"] = true
return nil
}
}
// ExecPath is a Chrome command line option to set the exec path.
// ExecPath is a command line option to set the exec path.
func ExecPath(path string) CommandLineOption {
return Flag("exec-path", path)
}
// Port is the Chrome command line option to set the remote debugging port.
func Port(port int) CommandLineOption {
return Flag("remote-debugging-port", port)
}
// UserDataDir is the Chrome command line option to set the user data dir.
// UserDataDir is the command line option to set the user data dir.
//
// Note: set this option to manually set the profile directory used by Chrome.
// When this is not set, then a default path will be created in the /tmp
@@ -370,27 +367,17 @@ func UserDataDir(dir string) CommandLineOption {
return Flag("user-data-dir", dir)
}
// StartURL is the Chrome command line option to set the initial URL.
func StartURL(urlstr string) CommandLineOption {
return Flag("start-url", urlstr)
}
// Proxy is the Chrome command line option to set the outbound proxy.
func Proxy(proxy string) CommandLineOption {
// ProxyServer is the command line option to set the outbound proxy server.
func ProxyServer(proxy string) CommandLineOption {
return Flag("proxy-server", proxy)
}
// ProxyPacURL is the Chrome command line option to set the URL of a proxy PAC file.
func ProxyPacURL(pacURL url.URL) CommandLineOption {
return Flag("proxy-pac-url", pacURL.String())
}
// WindowSize is the Chrome command line option to set the initial window size.
// WindowSize is the command line option to set the initial window size.
func WindowSize(width, height int) CommandLineOption {
return Flag("window-size", fmt.Sprintf("%d,%d", width, height))
}
// UserAgent is the Chrome command line option to set the default User-Agent
// UserAgent is the command line option to set the default User-Agent
// header.
func UserAgent(userAgent string) CommandLineOption {
return Flag("user-agent", userAgent)
@@ -413,45 +400,83 @@ func NoDefaultBrowserCheck(m map[string]interface{}) error {
return Flag("no-default-browser-check", true)(m)
}
// DisableGPU is the Chrome command line option to disable the GPU process.
// RemoteDebuggingPort is the command line option to set the remote
// debugging port.
func RemoteDebuggingPort(port int) CommandLineOption {
return Flag("remote-debugging-port", port)
}
// Headless is the command line option to run in headless mode.
func Headless(m map[string]interface{}) error {
return Flag("headless", true)(m)
}
// DisableGPU is the command line option to disable the GPU process.
func DisableGPU(m map[string]interface{}) error {
return Flag("disable-gpu", true)(m)
}
// CmdOpt is a Chrome command line option to modify the underlying exec.Cmd
// prior to invocation.
// URL is the command line option to add a URL to open on process start.
//
// Note: this can be specified multiple times, and each URL will be opened in a
// new tab.
func URL(urlstr string) CommandLineOption {
return func(m map[string]interface{}) error {
var urls []string
if u, ok := m["url-opts"]; ok {
urls, ok = u.([]string)
if !ok {
return ErrInvalidURLOpts
}
}
m["url-opts"] = append(urls, urlstr)
return nil
}
}
// CmdOpt is a command line option to modify the underlying exec.Cmd
// prior to the call to exec.Cmd.Start in Run.
func CmdOpt(o func(*exec.Cmd) error) CommandLineOption {
return func(m map[string]interface{}) error {
var opts []func(*exec.Cmd) error
if e, ok := m["cmd-opts"]; ok {
opts, ok = e.([]func(*exec.Cmd) error)
if !ok {
return errors.New("cmd-opts is in invalid state")
return ErrInvalidCmdOpts
}
}
m["cmd-opts"] = append(opts, o)
return nil
}
}
// ProcessOpt is a Chrome command line option to modify the child os.Process
// after started exec.Cmd.Start.
// ProcessOpt is a command line option to modify the child os.Process
// after the call to exec.Cmd.Start in Run.
func ProcessOpt(o func(*os.Process) error) CommandLineOption {
return func(m map[string]interface{}) error {
var opts []func(*os.Process) error
if e, ok := m["process-opts"]; ok {
opts, ok = e.([]func(*os.Process) error)
if !ok {
return errors.New("process-opts is in invalid state")
return ErrInvalidProcessOpts
}
}
m["process-opts"] = append(opts, o)
return nil
}
}
// LookChromeNames looks for the platform's DefaultChromeNames and any
// additional names using exec.LookPath, returning the first encountered
// location or the platform's DefaultChromePath if no names are found on the
// path.
func LookChromeNames(additional ...string) string {
for _, p := range append(additional, DefaultChromeNames...) {
path, err := exec.LookPath(p)
if err == nil {
return path
}
}
return DefaultChromePath
}

View File

@@ -24,21 +24,3 @@ func KillProcessGroup(m map[string]interface{}) error {
func ForceKill(m map[string]interface{}) error {
return nil
}
// EdgeDiagnosticsAdapterWithPath is a command line option to specify using the
// Microsoft Edge Diagnostics adapter at the specified path.
func EdgeDiagnosticsAdapterWithPathAndPort(path string, port int) CommandLineOption {
return func(m map[string]interface{}) error {
m["exec-path"] = path
m["port"] = port
return nil
}
}
// EdgeDiagnosticsAdapter is a command line option to specify using the
// Microsoft Edge Diagnostics adapter found on the path.
//
// If the
func EdgeDiagnosticsAdapter() CommandLineOption {
return EdgeDiagnosticsAdapterWithPathAndPort(findEdgePath(), 9222)
}

2
sel.go
View File

@@ -80,7 +80,7 @@ func (s *Selector) Do(ctxt context.Context, h cdp.Executor) error {
// are invalidated prior to finishing the selector's by, wait, check, and after
// funcs.
func (s *Selector) run(ctxt context.Context, h *TargetHandler) chan error {
ch := make(chan error)
ch := make(chan error, 1)
go func() {
defer close(ch)