Package org.apache.spark

Class SparkConf

Object

org.apache.spark.SparkConf

All Implemented Interfaces:

Serializable, Cloneable, org.apache.spark.internal.Logging, ReadOnlySparkConf

public class SparkConf
extends Object
implements ReadOnlySparkConf, Cloneable, org.apache.spark.internal.Logging, Serializable

Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.

Most of the time, you would create a SparkConf object with new SparkConf(), which will load values from any spark.* Java system properties set in your application as well. In this case, parameters you set directly on the SparkConf object take priority over system properties.

For unit tests, you can also call new SparkConf(false) to skip loading external settings and get the same configuration no matter what the system properties are.

All setter methods in this class support chaining. For example, you can write new SparkConf().setMaster("local").setAppName("My app").

param: loadDefaults whether to also load values from Java system properties

See Also:

• Serialized Form

Note:

Once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor	Description
SparkConf()	Create a SparkConf that loads defaults from system properties and the classpath
SparkConf(boolean loadDefaults)

Method Summary

Modifier and Type	Method	Description
SparkConf	clone()	Copy this object
boolean	contains(String key)	Does the configuration contain a given parameter?
scala.Tuple2<String,String>[]	getAll()	Get all parameters as a list of pairs
scala.Tuple2<String,String>[]	getAllWithPrefix(String prefix)	Get all parameters that start with prefix
String	getAppId()	Returns the Spark application id, valid in the Driver after TaskScheduler registration and from the start in the Executor.
scala.collection.immutable.Map<Object,String>	getAvroSchema()	Gets all the avro schemas in the configuration used in the generic Avro record serializer
static scala.Option<String>	getDeprecatedConfig(String key, Map<String,String> conf)	Looks for available deprecated keys for the given config option, and return the first value available.
scala.collection.immutable.Seq<scala.Tuple2<String,String>>	getExecutorEnv()	Get all executor environment variables set on this SparkConf
scala.Option<String>	getOption(String key)	Get a parameter as an Option
static boolean	isExecutorStartupConf(String name)	Return whether the given config should be passed to an executor on start-up.
static boolean	isSparkPortConf(String name)	Return true if the given config matches either spark.*.port or spark.port.*.
static void	logDeprecationWarning(String key)	Logs a warning message if the given config key is deprecated.
static org.apache.spark.internal.Logging.LogStringContext	LogStringContext(scala.StringContext sc)
static org.slf4j.Logger	org$apache$spark$internal$Logging$$log_()
static void	org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
SparkConf	registerAvroSchemas(scala.collection.immutable.Seq<org.apache.avro.Schema> schemas)	Use Kryo serialization and register the given set of Avro schemas so that the generic record serializer can decrease network IO
SparkConf	registerKryoClasses(Class<?>[] classes)	Use Kryo serialization and register the given set of classes with Kryo.
SparkConf	remove(String key)	Remove a parameter from the configuration
SparkConf	set(String key, String value)	Set a configuration variable.
SparkConf	setAll(scala.collection.Iterable<scala.Tuple2<String,String>> settings)	Set multiple parameters together
SparkConf	setAppName(String name)	Set a name for your application.
SparkConf	setExecutorEnv(String variable, String value)	Set an environment variable to be used when launching executors for this application.
SparkConf	setExecutorEnv(scala.collection.immutable.Seq<scala.Tuple2<String,String>> variables)	Set multiple environment variables to be used when launching executors.
SparkConf	setExecutorEnv(scala.Tuple2<String,String>[] variables)	Set multiple environment variables to be used when launching executors.
SparkConf	setIfMissing(String key, String value)	Set a parameter if it isn't already configured
SparkConf	setJars(String[] jars)	Set JAR files to distribute to the cluster.
SparkConf	setJars(scala.collection.immutable.Seq<String> jars)	Set JAR files to distribute to the cluster.
SparkConf	setMaster(String master)	The master URL to connect to, such as "local" to run locally with one thread, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.
SparkConf	setSparkHome(String home)	Set the location where Spark is installed on worker nodes.
String	toDebugString()	Return a string listing all keys and values, one per line.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

Methods inherited from interface org.apache.spark.ReadOnlySparkConf

catchIllegalValue, contains, get, get, getBoolean, getDouble, getInt, getLong, getSizeAsBytes, getSizeAsBytes, getSizeAsBytes, getSizeAsGb, getSizeAsGb, getSizeAsKb, getSizeAsKb, getSizeAsMb, getSizeAsMb, getTimeAsMs, getTimeAsMs, getTimeAsSeconds, getTimeAsSeconds

Constructor Details

SparkConf

public SparkConf(boolean loadDefaults)

SparkConf

public SparkConf()

Create a SparkConf that loads defaults from system properties and the classpath

Method Details

isExecutorStartupConf

public static boolean isExecutorStartupConf(String name)

Return whether the given config should be passed to an executor on start-up.

Certain authentication configs are required from the executor when it connects to the scheduler, while the rest of the spark configs can be inherited from the driver later.

Parameters:

name - (undocumented)

Returns:

(undocumented)

isSparkPortConf

public static boolean isSparkPortConf(String name)

Return true if the given config matches either spark.*.port or spark.port.*.

Parameters:

name - (undocumented)

Returns:

(undocumented)

getDeprecatedConfig

public static scala.Option<String> getDeprecatedConfig(String key,
 Map<String,String> conf)

Looks for available deprecated keys for the given config option, and return the first value available.

Parameters:

key - (undocumented)

conf - (undocumented)

Returns:

(undocumented)

logDeprecationWarning

public static void logDeprecationWarning(String key)

Logs a warning message if the given config key is deprecated.

Parameters:

key - (undocumented)

org$apache$spark$internal$Logging$$log_

public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

org$apache$spark$internal$Logging$$log__$eq

public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)

LogStringContext

public static org.apache.spark.internal.Logging.LogStringContext LogStringContext(scala.StringContext sc)

set

public SparkConf set(