com.spinn3r.api
Class BaseClient

java.lang.Object
  com.spinn3r.api.BaseClient

All Implemented Interfaces:

Client

Direct Known Subclasses:

CommentClient, CommentHistoryClient, FeedClient, FeedEntryClient, FeedHistoryClient, LinkClient, LinkHistoryClient, PermalinkClient, PermalinkEntryClient, PermalinkHistoryClient, PermalinkStatusClient, SourceClient, SourceListClient

public abstract class BaseClient
extends java.lang.Object
implements Client

Generic client support used which need to be in all APIs. See Main.java for usage example. All implementations need to catch and handle Exceptions.

Restarting

When stopping/starting the API you need to persist an 'after' point where to start the URL from again. This is just the timestamp of the most recent URL you found minus a buffer.

Field Summary

static java.lang.String	ACCEPT_ENCODING_HEADER
static java.lang.String	COMMENT_HANDLER
protected Config	config
static int	DEFAULT_CONNECT_TIMEOUT These properties specify the default connect and read timeout (resp.) for the protocol handler used by java.net.URLConnection.
static java.lang.String	DEFAULT_HOST Default hostname for building the router URL.
static boolean	DEFAULT_HTTP_KEEPALIVE Whether we should use HTTP Keep Alive in java.net.URL.
static int	DEFAULT_MAX_REDIRECTS Specify the maximum number of redirects to use.
static int	DEFAULT_READ_TIMEOUT Lower read timeout.
protected boolean	disable_parse
static java.lang.String	FEED_HANDLER
static java.lang.String	GZIP_ENCODING
protected boolean	isCompressed True if the last API call was compressed.
static java.lang.String	LINK_HANDLER
static int	NETWORKADDRESS_CACHE_TTL Specified in java.security to indicate the caching policy for successful name lookups from the name service..
static java.lang.String	NS_API
static java.lang.String	NS_ATOM
static java.lang.String	NS_COMMENT
static java.lang.String	NS_DC
static java.lang.String	NS_FEED
static java.lang.String	NS_LINK
static java.lang.String	NS_POST
static java.lang.String	NS_SOURCE
static java.lang.String	NS_TARGET
static java.lang.String	NS_WEBLOG
static java.lang.String	PERMALINK_HANDLER
static int	RESTART_BUFFER Go back in time to make sure we recrawl everything.
protected java.util.List	results
static long	RETRY_MAX Maximum number of retries.
static java.lang.String	USER_AGENT
static java.lang.String	USER_AGENT_HEADER

Constructor Summary

BaseClient()

Method Summary

static void	addParam(java.lang.StringBuffer buff, java.lang.String name, java.lang.Object value)
static void	addParam(java.lang.StringBuffer buff, java.lang.String name, java.lang.Object value, boolean optional)
static void	addParam(java.lang.StringBuffer buff, java.lang.String name, java.lang.Object value, boolean optional, boolean urlencode) Add a parameter to the first request URL.
ContentApi.Response	doProtobufFetch(java.lang.String resource)
org.w3c.dom.Document	doXmlFetch(java.lang.String resource)
boolean	empty(java.lang.String value)
void	fetch(Config config)
protected java.lang.String	generateFirstRequestURL() Generate the first request URL based just on configuration directives.
long	getCallDuration() Get the value of callDuration.
Config	getConfig() Get the value of config.
protected int	getConservativeLimit() Conservative limit for items which should work in all situations (but might be slower)
static org.w3c.dom.Element	getElementByTagName(org.w3c.dom.Element current, java.lang.String name)
static org.w3c.dom.Element	getElementByTagName(org.w3c.dom.Element current, java.lang.String name, java.lang.String namespace)
static java.lang.String	getElementCDATAByTagName(org.w3c.dom.Element current, java.lang.String name)
static java.lang.String	getElementCDATAByTagName(org.w3c.dom.Element current, java.lang.String name, java.lang.String namespace)
java.lang.String	getHost() Get the host name for API calls.
java.io.InputStream	getInputStream() Get the InputStream for dealing with the XML of the API directly.
java.lang.String	getLastRequestURL() Get the last requested URL for debug and logging purposes.
int	getLimit() Return the correct limit, factoring in the limit set by the user.
protected int	getMaxLimit() Return the maximum number of request per call.
java.lang.String	getNextRequestURL() Get the value of nextRequestURL.
static java.util.Map<java.lang.String,java.lang.String>	getopt(java.lang.String[] args) Parse command line arguments like --foo=bar where foo is the key and bar is the value.
protected int	getOptimalLimit() Return the optimal limit for fetches.
long	getParseDuration() Get the value of parseDuration.
java.util.Date	getRestartPoint() When the API needs to shutdown you need to call this method FIRST and persist it.
abstract java.lang.String	getRouter() Return the router for this client.
long	getSleepDuration() Get the value of sleepDuration.
boolean	hasMoreResults() Return true if more results are available.
protected static void	padd(int v, java.lang.StringBuffer buff)
static java.util.List	parseChildNodesAsList(org.w3c.dom.Element current, java.lang.String name)
static float	parseFloat(java.lang.String v, float _default)
static int	parseInt(java.lang.String v)
protected abstract BaseResult	parseItem(ContentApi.Entry current)
protected BaseResult	parseItem(ContentApi.Entry entry, BaseResult result)
protected BaseResult	parseItem(org.w3c.dom.Element current) Parse an individual item which might be specific to this client.
protected BaseResult	parseItem(org.w3c.dom.Element current, BaseResult result)
static long	parseLong(java.lang.String v)
static java.util.List	parseTags(org.w3c.dom.Element current)
protected void	protobufParse(ContentApi.Response response) We've received a response from the API so parse it out.
void	setCallDuration(long callDuration) Set the value of callDuration.
void	setConfig(Config config) Set the value of config.
void	setHost(java.lang.String v) Set the host for API alls.
void	setLastRequestURL(java.lang.String lastRequestURL) Set the value of lastRequestURL.
void	setNextRequestURL(java.lang.String next) Set the value of nextRequestURL.
protected java.lang.String	setParam(java.lang.String v, java.lang.String key, java.lang.Object value) Set a parameter in the HTTP URL.
void	setParseDuration(long parseDuration) Set the value of parseDuration.
void	setSleepDuration(long sleepDuration) Set the value of sleepDuration.
static java.lang.String	toISO8601(java.util.Date date) Return a date to an ISO 8601 value for specifying to the URL with an 'after' param.
protected void	xmlParse(org.w3c.dom.Document doc) We've received a response from the API so parse it out.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.spinn3r.api.Client

fetch, getResults

Field Detail

RETRY_MAX

public static long RETRY_MAX

Maximum number of retries.

RESTART_BUFFER

public static final int RESTART_BUFFER

Go back in time to make sure we recrawl everything.

See Also:

Constant Field Values

NS_API

public static final java.lang.String NS_API

See Also:

Constant Field Values

NS_DC

public static final java.lang.String NS_DC

See Also:

Constant Field Values

NS_ATOM

public static final java.lang.String NS_ATOM

See Also:

Constant Field Values

NS_WEBLOG

public static final java.lang.String NS_WEBLOG

See Also:

Constant Field Values

NS_SOURCE

public static final java.lang.String NS_SOURCE

See Also:

Constant Field Values

NS_POST

public static final java.lang.String NS_POST

See Also:

Constant Field Values

NS_FEED

public static final java.lang.String NS_FEED

See Also:

Constant Field Values

NS_LINK

public static final java.lang.String NS_LINK

See Also:

Constant Field Values

NS_TARGET

public static final java.lang.String NS_TARGET

See Also:

Constant Field Values

NS_COMMENT

public static final java.lang.String NS_COMMENT

See Also:

Constant Field Values

USER_AGENT_HEADER

public static final java.lang.String USER_AGENT_HEADER

See Also:

Constant Field Values

ACCEPT_ENCODING_HEADER

public static final java.lang.String ACCEPT_ENCODING_HEADER

See Also:

Constant Field Values

FEED_HANDLER

public static java.lang.String FEED_HANDLER

PERMALINK_HANDLER

public static java.lang.String PERMALINK_HANDLER

COMMENT_HANDLER

public static java.lang.String COMMENT_HANDLER

LINK_HANDLER

public static java.lang.String LINK_HANDLER

GZIP_ENCODING

public static final java.lang.String GZIP_ENCODING

See Also:

Constant Field Values

USER_AGENT

public static java.lang.String USER_AGENT

DEFAULT_HOST

public static java.lang.String DEFAULT_HOST

Default hostname for building the router URL. This can be changed to use an additional dedicated host if necessary.

NETWORKADDRESS_CACHE_TTL

public static int NETWORKADDRESS_CACHE_TTL

Specified in java.security to indicate the caching policy for successful name lookups from the name service.. The value is specified as as integer to indicate the number of seconds to cache the successful lookup. sun.net.inetaddr.ttl: This is a sun private system property which corresponds to networkaddress.cache.ttl. It takes the same value and has the same meaning, but can be set as a command-line option. However, the preferred way is to use the security property mentioned above. A value of -1 indicates "cache forever".

DEFAULT_CONNECT_TIMEOUT

public static int DEFAULT_CONNECT_TIMEOUT

These properties specify the default connect and read timeout (resp.) for the protocol handler used by java.net.URLConnection. sun.net.client.defaultConnectTimeout specifies the timeout (in milliseconds) to establish the connection to the host. For example for http connections it is the timeout when establishing the connection to the http server. For ftp connection it is the timeout when establishing the connection to ftp servers. sun.net.client.defaultReadTimeout specifies the timeout (in milliseconds) when reading from input stream when a connection is established to a resource.

DEFAULT_READ_TIMEOUT

public static int DEFAULT_READ_TIMEOUT

Lower read timeout. Makes NO sense to wait for five minutes to read a byte from spinn3r.

DEFAULT_MAX_REDIRECTS

public static int DEFAULT_MAX_REDIRECTS

Specify the maximum number of redirects to use.

DEFAULT_HTTP_KEEPALIVE

public static boolean DEFAULT_HTTP_KEEPALIVE

Whether we should use HTTP Keep Alive in java.net.URL. We default to true here because MOST of our TCP connections WILL be used again since everything is to spinn3r.com.

disable_parse

protected boolean disable_parse

results

protected java.util.List results

config

protected Config config

isCompressed

protected boolean isCompressed

True if the last API call was compressed.

Constructor Detail

BaseClient

public BaseClient()

Method Detail

getInputStream

public java.io.InputStream getInputStream()
                                   throws java.io.IOException

Get the InputStream for dealing with the XML of the API directly. This is a LOCAL input stream so once fetch() has been called you can call this API multiple times and you're reading from a local buffer.

Specified by:

getInputStream in interface Client

Throws:

java.io.IOException

fetch

public void fetch(Config config)
           throws java.io.IOException,
                  ParseException,
                  java.lang.InterruptedException

Throws:

java.io.IOException

ParseException

java.lang.InterruptedException

doProtobufFetch

public ContentApi.Response doProtobufFetch(java.lang.String resource)
                                    throws java.io.IOException,
                                           java.lang.InterruptedException

Throws:

java.io.IOException

java.lang.InterruptedException

doXmlFetch

public org.w3c.dom.Document doXmlFetch(java.lang.String resource)
                                throws java.io.IOException,
                                       ParseException,
                                       java.lang.InterruptedException

Throws:

java.io.IOException

ParseException

java.lang.InterruptedException

xmlParse

protected void xmlParse(org.w3c.dom.Document doc)
                 throws java.lang.Exception

We've received a response from the API so parse it out.

Throws:

java.lang.Exception

protobufParse

protected void protobufParse(ContentApi.Response response)
                      throws java.lang.Exception

We've received a response from the API so parse it out.

Throws:

java.lang.Exception

generateFirstRequestURL

protected java.lang.String generateFirstRequestURL()

Generate the first request URL based just on configuration directives.

getMaxLimit

protected int getMaxLimit()

Return the maximum number of request per call. If the API has more content this might cause an error on our end if the limit is more than 10 or so.

getOptimalLimit

protected int getOptimalLimit()

Return the optimal limit for fetches. This is used to boost performance in some situations. We have to resort to the conservative limit if the HTTP server can't handle the result set size.

getConservativeLimit

protected int getConservativeLimit()

Conservative limit for items which should work in all situations (but might be slower)

getRouter

public abstract java.lang.String getRouter()

Return the router for this client. Right now this is either: http://api.spinn3r.com/rss/feed.getDelta? or http://api.spinn3r.com/rss/permalink.getDelta?

Specified by:

getRouter in interface Client

parseItem

protected BaseResult parseItem(org.w3c.dom.Element current)
                        throws java.lang.Exception

Parse an individual item which might be specific to this client.

Throws:

java.lang.Exception

parseItem

protected abstract BaseResult parseItem(ContentApi.Entry current)
                                 throws java.lang.Exception

Throws:

java.lang.Exception

parseItem

protected BaseResult parseItem(org.w3c.dom.Element current,
                               BaseResult result)
                        throws java.lang.Exception

Throws:

java.lang.Exception

parseItem

protected BaseResult parseItem(ContentApi.Entry entry,
                               BaseResult result)
                        throws java.lang.Exception

Throws:

java.lang.Exception

empty

public boolean empty(java.lang.String value)

parseInt

public static int parseInt(java.lang.String v)

parseLong

public static long parseLong(java.lang.String v)

parseFloat

public static float parseFloat(java.lang.String v,
                               float _default)

parseTags

public static java.util.List parseTags(org.w3c.dom.Element current)

parseChildNodesAsList

public static java.util.List parseChildNodesAsList(org.w3c.dom.Element current,
                                                   java.lang.String name)

getElementByTagName

public static org.w3c.dom.Element getElementByTagName(org.w3c.dom.Element current,
                                                      java.lang.String name)

getElementByTagName

public static org.w3c.dom.Element getElementByTagName(org.w3c.dom.Element current,
                                                      java.lang.String name,
                                                      java.lang.String namespace)

getElementCDATAByTagName

public static java.lang.String getElementCDATAByTagName(org.w3c.dom.Element current,
                                                        java.lang.String name)

getElementCDATAByTagName

public static java.lang.String getElementCDATAByTagName(org.w3c.dom.Element current,
                                                        java.lang.String name,
                                                        java.lang.String namespace)

toISO8601

public static java.lang.String toISO8601(java.util.Date date)

Return a date to an ISO 8601 value for specifying to the URL with an 'after' param.

padd

protected static void padd(int v,
                           java.lang.StringBuffer buff)

addParam

public static void addParam(java.lang.StringBuffer buff,
                            java.lang.String name,
                            java.lang.Object value)

addParam

public static void addParam(java.lang.StringBuffer buff,
                            java.lang.String name,
                            java.lang.Object value,
                            boolean optional)

addParam

public static void addParam(java.lang.StringBuffer buff,
                            java.lang.String name,
                            java.lang.Object value,
                            boolean optional,
                            boolean urlencode)

Add a parameter to the first request URL. After the first call this is no longer needed.

setParam

protected java.lang.String setParam(java.lang.String v,
                                    java.lang.String key,
                                    java.lang.Object value)

Set a parameter in the HTTP URL.

getLastRequestURL

public java.lang.String getLastRequestURL()

Get the last requested URL for debug and logging purposes.

Specified by:

getLastRequestURL in interface Client

setLastRequestURL

public void setLastRequestURL(java.lang.String lastRequestURL)

Set the value of lastRequestURL.

getNextRequestURL

public java.lang.String getNextRequestURL()

Get the value of nextRequestURL.

Specified by:

getNextRequestURL in interface Client

setNextRequestURL

public void setNextRequestURL(java.lang.String next)

Set the value of nextRequestURL.

getConfig

public Config getConfig()

Get the value of config.

Specified by:

getConfig in interface Client

setConfig

public void setConfig(Config config)

Set the value of config.

Specified by:

setConfig in interface Client

getCallDuration

public long getCallDuration()

Get the value of callDuration.

Specified by:

getCallDuration in interface Client

setCallDuration

public void setCallDuration(long callDuration)

Set the value of callDuration.

getSleepDuration

public long getSleepDuration()

Get the value of sleepDuration.

Specified by:

getSleepDuration in interface Client

setSleepDuration

public void setSleepDuration(long sleepDuration)

Set the value of sleepDuration.

Specified by:

setSleepDuration in interface Client

getParseDuration

public long getParseDuration()

Get the value of parseDuration.

Specified by:

getParseDuration in interface Client

setParseDuration

public void setParseDuration(long parseDuration)

Set the value of parseDuration.

Specified by:

setParseDuration in interface Client

setHost

public void setHost(java.lang.String v)

Set the host for API alls.

Specified by:

setHost in interface Client

getHost

public java.lang.String getHost()

Description copied from interface: Client

Get the host name for API calls.

Specified by:

getHost in interface Client

getRestartPoint

public java.util.Date getRestartPoint()

When the API needs to shutdown you need to call this method FIRST and persist it. Then when the API starts you need to call config.setAfter() with this value.

Specified by:

getRestartPoint in interface Client

getLimit

public int getLimit()

Return the correct limit, factoring in the limit set by the user.

hasMoreResults

public boolean hasMoreResults()

Return true if more results are available.

getopt

public static java.util.Map<java.lang.String,java.lang.String> getopt(java.lang.String[] args)

Parse command line arguments like --foo=bar where foo is the key and bar is the value.