python/sqlglot
Fork 0
Code Issues Activity

Merging upstream version 28.6.0.

Browse source 
Signed-off-by: Daniel Baumann <daniel@debian.org>
This commit is contained in:
Daniel Baumann 2026-01-15 10:55:13 +01:00
commit 257c5ef672
Signed by: daniel
GPG key ID: FBB4F0E80A80222F
66 changed files with 3799 additions and 447 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.github/scripts/get_integration_test_params.py vendored
View file

@ -113,7 +113,7 @@ if __name__ == "__main__":
print(f"Handling event: \n" + json.dumps(event, indent=2))
# for pull_request events, the body is located at github.event.pull_request.body
pr_description: str = event.get("pull_request", {}).get("body", "")
pr_description: str = event.get("pull_request", {}).get("body") or ""
dialects = []
should_run = False

2
.github/workflows/run-integration-tests.yml vendored
View file

@ -57,7 +57,7 @@ jobs:
name: Run Integration Tests
runs-on: ubuntu-latest
needs: should-run
if: needs.should-run.outputs.skip == 'false' && github.event.repository.fork != true
if: needs.should-run.outputs.skip == 'false' && github.event.pull_request.head.repo.full_name == github.repository
steps:
- name: Acquire credentials
id: app-token

36
CHANGELOG.md
View file

@ -1,6 +1,41 @@
Changelog
=========
## [v28.5.0] - 2025-12-17
### :boom: BREAKING CHANGES
- due to [`4dfc810`](https://github.com/tobymao/sqlglot/commit/4dfc810f45d5a617ada2ba4ed57002549c8d1853) - support transpilation of BOOLNOT from snowflake to duckdb *(PR [#6577](https://github.com/tobymao/sqlglot/pull/6577) by [@fivetran-felixhuang](https://github.com/fivetran-felixhuang))*:
support transpilation of BOOLNOT from snowflake to duckdb (#6577)
- due to [`b857185`](https://github.com/tobymao/sqlglot/commit/b8571850ca55802671484d118560a7b90e893c39) - remove Sysdate in favor of CurrentTimestamp with sysdate arg *(PR [#6584](https://github.com/tobymao/sqlglot/pull/6584) by [@georgesittas](https://github.com/georgesittas))*:
remove Sysdate in favor of CurrentTimestamp with sysdate arg (#6584)
- due to [`bf217d6`](https://github.com/tobymao/sqlglot/commit/bf217d69f92efcbce5b69d637976e915ca63998d) - make `JSONArrayAgg` an `AggFunc` *(PR [#6585](https://github.com/tobymao/sqlglot/pull/6585) by [@AbhishekASLK](https://github.com/AbhishekASLK))*:
make `JSONArrayAgg` an `AggFunc` (#6585)
- due to [`604efe5`](https://github.com/tobymao/sqlglot/commit/604efe5cf5812d0b1dd9d625ed278907d0d7fb8f) - Type annotation fixes for TO_TIMESTAMP* *(PR [#6557](https://github.com/tobymao/sqlglot/pull/6557) by [@fivetran-kwoodbeck](https://github.com/fivetran-kwoodbeck))*:
Type annotation fixes for TO_TIMESTAMP* (#6557)
### :sparkles: New Features
- [`4dfc810`](https://github.com/tobymao/sqlglot/commit/4dfc810f45d5a617ada2ba4ed57002549c8d1853) - **snowflake**: support transpilation of BOOLNOT from snowflake to duckdb *(PR [#6577](https://github.com/tobymao/sqlglot/pull/6577) by [@fivetran-felixhuang](https://github.com/fivetran-felixhuang))*
- [`7077981`](https://github.com/tobymao/sqlglot/commit/707798166c1b45e633bd0e8d02d1c0146598b03a) - **snowflake**: Transpilation of Snowflake MONTHS_BETWEEN to DuckDB *(PR [#6561](https://github.com/tobymao/sqlglot/pull/6561) by [@fivetran-kwoodbeck](https://github.com/fivetran-kwoodbeck))*
- [`604efe5`](https://github.com/tobymao/sqlglot/commit/604efe5cf5812d0b1dd9d625ed278907d0d7fb8f) - **snowflake**: Type annotation fixes for TO_TIMESTAMP* *(PR [#6557](https://github.com/tobymao/sqlglot/pull/6557) by [@fivetran-kwoodbeck](https://github.com/fivetran-kwoodbeck))*
- [`3567880`](https://github.com/tobymao/sqlglot/commit/35678808dafb37c5d37c806682e6af9b6351bced) - add tokens to functions *(commit by [@tobymao](https://github.com/tobymao))*
### :bug: Bug Fixes
- [`b857185`](https://github.com/tobymao/sqlglot/commit/b8571850ca55802671484d118560a7b90e893c39) - **snowflake**: remove Sysdate in favor of CurrentTimestamp with sysdate arg *(PR [#6584](https://github.com/tobymao/sqlglot/pull/6584) by [@georgesittas](https://github.com/georgesittas))*
- [`bf217d6`](https://github.com/tobymao/sqlglot/commit/bf217d69f92efcbce5b69d637976e915ca63998d) - make `JSONArrayAgg` an `AggFunc` *(PR [#6585](https://github.com/tobymao/sqlglot/pull/6585) by [@AbhishekASLK](https://github.com/AbhishekASLK))*
- [`48f5e99`](https://github.com/tobymao/sqlglot/commit/48f5e999d3d3f6ad51c30e7a33a3a574d0e50d2b) - **duckdb**: preserve l/r-trim syntax *(PR [#6588](https://github.com/tobymao/sqlglot/pull/6588) by [@georgesittas](https://github.com/georgesittas))*
- :arrow_lower_right: *fixes issue [#6587](https://github.com/tobymao/sqlglot/issues/6587) opened by [@baruchoxman](https://github.com/baruchoxman)*
### :wrench: Chores
- [`ea0263a`](https://github.com/tobymao/sqlglot/commit/ea0263aa555591b03b06a4b6dee093fe42b545f9) - Skip integration tests GA for external contributors & fix `git diff` *(PR [#6582](https://github.com/tobymao/sqlglot/pull/6582) by [@VaggelisD](https://github.com/VaggelisD))*
## [v28.4.1] - 2025-12-16
### :boom: BREAKING CHANGES
- due to [`cfc9346`](https://github.com/tobymao/sqlglot/commit/cfc9346ba0477523d3de8f923d83fd09814b22ac) - bump sqlglotrs to 0.10.0 *(commit by [@tobymao](https://github.com/tobymao))*:
@ -11516,3 +11551,4 @@ Changelog
[v28.3.0]: https://github.com/tobymao/sqlglot/compare/v28.2.0...v28.3.0
[v28.4.0]: https://github.com/tobymao/sqlglot/compare/v27.6.1...v28.4.0
[v28.4.1]: https://github.com/tobymao/sqlglot/compare/v28.4.0...v28.4.1
[v28.5.0]: https://github.com/tobymao/sqlglot/compare/v28.4.1...v28.5.0

42
README.md
View file

@ -32,6 +32,7 @@ Contributions are very welcome in SQLGlot; read the [contribution guide](https:/
* [Used By](#used-by)
* [Documentation](#documentation)
* [Run Tests and Lint](#run-tests-and-lint)
* [Deployment](#deployment)
* [Benchmarks](#benchmarks)
* [Optional Dependencies](#optional-dependencies)
* [Supported Dialects](#supported-dialects)
@ -526,6 +527,21 @@ make test # Unit and integration tests (or test-rs, to use the Rust tokenizer)
make check # Full test suite & linter checks
```
## Deployment
To deploy a new SQLGlot version, follow these steps:
1. Run `git pull` to make sure the local git repo is at the head of the main branch
2. If the Rust tokenizer code changed since the last version release:
1. Bump the `version` attribute under the `package` header in `sqlglotrs/Cargo.toml`
2. Run `make install-dev`. This will update the `Cargo.lock` file
3. Commit the changes made to `Cargo.toml` and `Cargo.lock`
3. Do a `git tag` operation to bump the SQLGlot version, e.g. `git tag v28.5.0`
4. Run `git push && git push --tags` to deploy the new version
> [!IMPORTANT]
> If there are any breaking changes since the last version release, make sure to deploy either a minor or major version for both sqlglot and sqlglotrs. Refer to SQLGlot's [versioning scheme](#versioning) for more information.
## Benchmarks
[Benchmarks](https://github.com/tobymao/sqlglot/blob/main/benchmarks/bench.py) run on Python 3.10.12 in seconds.
@ -588,3 +604,29 @@ x + interval '1' month
**Official Dialects** are maintained by the core SQLGlot team with higher priority for bug fixes and feature additions.
**Community Dialects** are developed and maintained primarily through community contributions. These are fully functional but may receive lower priority for issue resolution compared to officially supported dialects. We welcome and encourage community contributions to improve these dialects.
### Creating a Dialect Plugin
If your database isn't supported, you can create a plugin that registers a custom dialect via entry points. Create a package with your dialect class and register it in `setup.py`:
```python
from setuptools import setup
setup(
name="mydb-sqlglot-dialect",
entry_points={
"sqlglot.dialects": [
"mydb = my_package.dialect:MyDB",
],
},
)
```
The dialect will be automatically discovered and can be used like any built-in dialect:
```python
from sqlglot import transpile
transpile("SELECT * FROM t", read="mydb", write="postgres")
```
See the [Custom Dialects](#custom-dialects) section for implementation details.

2
posts/onboarding.md
View file

@ -7,7 +7,7 @@
- [Matching text](#matching-text)
- [Utility methods](#utility-methods)
- [Command fallback](#command-fallback)
- [Expression parsing](#expression-parsing)
- [Dialect-specific parsing](#dialect-specific-parsing)
- [Generator](#generator)
- [Generating queries](#generating-queries)
- [Utility methods](#utility-methods-1)

12
sqlglot/dialects/bigquery.py
View file

@ -51,6 +51,8 @@ JSON_EXTRACT_TYPE = t.Union[exp.JSONExtract, exp.JSONExtractScalar, exp.JSONExtr
DQUOTES_ESCAPING_JSON_FUNCTIONS = ("JSON_QUERY", "JSON_VALUE", "JSON_QUERY_ARRAY")
MAKE_INTERVAL_KWARGS = ["year", "month", "day", "hour", "minute", "second"]
def _derived_table_values_to_unnest(self: BigQuery.Generator, expression: exp.Values) -> str:
if not expression.find_ancestor(exp.From, exp.Join):
@ -389,7 +391,9 @@ class BigQuery(Dialect):
EXCLUDES_PSEUDOCOLUMNS_FROM_STAR = True
QUERY_RESULTS_ARE_STRUCTS = True
JSON_EXTRACT_SCALAR_SCALAR_ONLY = True
LEAST_GREATEST_IGNORES_NULLS = False
DEFAULT_NULL_TYPE = exp.DataType.Type.BIGINT
PRIORITIZE_NON_LITERAL_TYPES = True
# https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#initcap
INITCAP_DEFAULT_DELIMITER_CHARS = ' \t\n\r\f\v\\[\\](){}/|<>!?@"^#$&~_,.:;*%+\\-'
@ -602,12 +606,6 @@ class BigQuery(Dialect):
"EDIT_DISTANCE": _build_levenshtein,
"FORMAT_DATE": _build_format_time(exp.TsOrDsToDate),
"GENERATE_ARRAY": exp.GenerateSeries.from_arg_list,
"GREATEST": lambda args: exp.Greatest(
this=seq_get(args, 0), expressions=args[1:], null_if_any_null=True
),
"LEAST": lambda args: exp.Least(
this=seq_get(args, 0), expressions=args[1:], null_if_any_null=True
),
"JSON_EXTRACT_SCALAR": _build_extract_json_with_default_path(exp.JSONExtractScalar),
"JSON_EXTRACT_ARRAY": _build_extract_json_with_default_path(exp.JSONExtractArray),
"JSON_EXTRACT_STRING_ARRAY": _build_extract_json_with_default_path(exp.JSONValueArray),
@ -964,7 +962,7 @@ class BigQuery(Dialect):
def _parse_make_interval(self) -> exp.MakeInterval:
expr = exp.MakeInterval()
for arg_key in expr.arg_types:
for arg_key in MAKE_INTERVAL_KWARGS:
value = self._parse_lambda()
if not value:

2
sqlglot/dialects/clickhouse.py
View file

@ -565,6 +565,8 @@ class ClickHouse(Dialect):
"MEDIAN": lambda self: self._parse_quantile(),
"COLUMNS": lambda self: self._parse_columns(),
"TUPLE": lambda self: exp.Struct.from_arg_list(self._parse_function_args(alias=True)),
"AND": lambda self: exp.and_(*self._parse_function_args(alias=False)),
"OR": lambda self: exp.or_(*self._parse_function_args(alias=False)),
}
FUNCTION_PARSERS.pop("MATCH")

91
sqlglot/dialects/dialect.py
View file

@ -27,6 +27,8 @@ from sqlglot.tokens import Token, Tokenizer, TokenType
from sqlglot.trie import new_trie
from sqlglot.typing import EXPRESSION_METADATA
from importlib.metadata import entry_points
DATE_ADD_OR_DIFF = t.Union[
exp.DateAdd,
exp.DateDiff,
@ -66,6 +68,8 @@ UNESCAPED_SEQUENCES = {
"\\\\": "\\",
}
PLUGIN_GROUP_NAME = "sqlglot.dialects"
class Dialects(str, Enum):
"""Dialects supported by SQLGLot."""
@ -153,12 +157,54 @@ class _Dialect(type):
if isinstance(key, Dialects):
key = key.value
# This import will lead to a new dialect being loaded, and hence, registered.
# We check that the key is an actual sqlglot module to avoid blindly importing
# files. Custom user dialects need to be imported at the top-level package, in
# order for them to be registered as soon as possible.
# 1. Try standard sqlglot modules first
if key in DIALECT_MODULE_NAMES:
module = importlib.import_module(f"sqlglot.dialects.{key}")
# If module was already imported, the class may not be in _classes
# Find and register the dialect class from the module
if key not in cls._classes:
for attr_name in dir(module):
attr = getattr(module, attr_name, None)
if (
isinstance(attr, type)
and issubclass(attr, Dialect)
and attr.__name__.lower() == key
):
cls._classes[key] = attr
break
return
# 2. Try entry points (for plugins)
try:
all_eps = entry_points()
# Python 3.10+ has select() method, older versions use dict-like access
if hasattr(all_eps, "select"):
eps = all_eps.select(group=PLUGIN_GROUP_NAME, name=key)
else:
# For older Python versions, entry_points() returns a dict-like object
group_eps = all_eps.get(PLUGIN_GROUP_NAME, []) # type: ignore
eps = [ep for ep in group_eps if ep.name == key] # type: ignore
for entry_point in eps:
dialect_class = entry_point.load()
# Verify it's a Dialect subclass
# issubclass() returns False if not a subclass, TypeError only if not a class at all
if isinstance(dialect_class, type) and issubclass(dialect_class, Dialect):
# Register the dialect using the entry point name (key)
# The metaclass may have registered it by class name, but we need it by entry point name
if key not in cls._classes:
cls._classes[key] = dialect_class
return
except ImportError:
# entry_point.load() failed (bad plugin - module/class doesn't exist)
pass
# 3. Try direct import (for backward compatibility)
# This allows namespace packages or explicit imports to work
try:
importlib.import_module(f"sqlglot.dialects.{key}")
except ImportError:
pass
@classmethod
def __getitem__(cls, key: str) -> t.Type[Dialect]:
@ -741,6 +787,18 @@ class Dialect(metaclass=_Dialect):
For example, in BigQuery the default type of the NULL value is INT64.
"""
LEAST_GREATEST_IGNORES_NULLS = True
"""
Whether LEAST/GREATEST functions ignore NULL values, e.g:
- BigQuery, Snowflake, MySQL, Presto/Trino: LEAST(1, NULL, 2) -> NULL
- Spark, Postgres, DuckDB, TSQL: LEAST(1, NULL, 2) -> 1
"""
PRIORITIZE_NON_LITERAL_TYPES = False
"""
Whether to prioritize non-literal types over literals during type annotation.
"""
# --- Autofilled ---
tokenizer_class = Tokenizer
@ -935,7 +993,9 @@ class Dialect(metaclass=_Dialect):
result = cls.get(dialect_name.strip())
if not result:
suggest_closest_match_and_fail("dialect", dialect_name, list(DIALECT_MODULE_NAMES))
# Include both built-in dialects and any loaded dialects for better error messages
all_dialects = set(DIALECT_MODULE_NAMES) | set(cls._classes.keys())
suggest_closest_match_and_fail("dialect", dialect_name, all_dialects)
assert result is not None
return result(**kwargs)
@ -2207,3 +2267,24 @@ def regexp_replace_global_modifier(expression: exp.RegexpReplace) -> exp.Express
modifiers = exp.Literal.string(value + "g")
return modifiers
def getbit_sql(self: Generator, expression: exp.Getbit) -> str:
"""
Generates SQL for Getbit according to DuckDB and Postgres, transpiling it if either:
1. The zero index corresponds to the least-significant bit
2. The input type is an integer value
"""
value = expression.this
position = expression.expression
if not expression.args.get("zero_is_msb") and expression.is_type(
*exp.DataType.SIGNED_INTEGER_TYPES, *exp.DataType.UNSIGNED_INTEGER_TYPES
):
# Use bitwise operations: (value >> position) & 1
shifted = exp.BitwiseRightShift(this=value, expression=position)
masked = exp.BitwiseAnd(this=shifted, expression=exp.Literal.number(1))
return self.sql(masked)
return self.func("GET_BIT", value, position)

1
sqlglot/dialects/doris.py
View file

@ -178,6 +178,7 @@ class Doris(MySQL):
VARCHAR_REQUIRES_SIZE = False
WITH_PROPERTIES_PREFIX = "PROPERTIES"
RENAME_TABLE_WITH_DB = False
UPDATE_STATEMENT_SUPPORTS_FROM = True
TYPE_MAPPING = {
**MySQL.Generator.TYPE_MAPPING,

1
sqlglot/dialects/druid.py
View file

@ -17,4 +17,5 @@ class Druid(Dialect):
**generator.Generator.TRANSFORMS,
exp.CurrentTimestamp: lambda *_: "CURRENT_TIMESTAMP",
exp.Mod: rename_func("MOD"),
exp.Array: lambda self, e: f"ARRAY[{self.expressions(e)}]",
}

975
sqlglot/dialects/duckdb.py
View file

File diff suppressed because it is too large Load diff

18
sqlglot/dialects/exasol.py
View file

@ -16,6 +16,7 @@ from sqlglot.dialects.dialect import (
build_date_delta,
no_last_day_sql,
DATE_ADD_OR_SUB,
build_timetostr_or_tochar,
)
from sqlglot.generator import unsupported_args
from sqlglot.helper import seq_get
@ -319,6 +320,7 @@ class Exasol(Dialect):
"ENDIF": TokenType.END,
"LONG VARCHAR": TokenType.TEXT,
"SEPARATOR": TokenType.SEPARATOR,
"SYSTIMESTAMP": TokenType.SYSTIMESTAMP,
}
KEYWORDS.pop("DIV")
@ -368,7 +370,7 @@ class Exasol(Dialect):
"TRUNCATE": _build_trunc,
"VAR_POP": exp.VariancePop.from_arg_list,
"APPROXIMATE_COUNT_DISTINCT": exp.ApproxDistinct.from_arg_list,
"TO_CHAR": build_formatted_time(exp.ToChar, "exasol"),
"TO_CHAR": build_timetostr_or_tochar,
"TO_DATE": build_formatted_time(exp.TsOrDsToDate, "exasol"),
# https://docs.exasol.com/db/latest/sql_references/functions/alphabeticallistfunctions/convert_tz.htm
"CONVERT_TZ": lambda args: exp.ConvertTimezone(
@ -387,6 +389,17 @@ class Exasol(Dialect):
this=self._match(TokenType.IS) and self._parse_string(),
),
}
FUNC_TOKENS = {
*parser.Parser.FUNC_TOKENS,
TokenType.SYSTIMESTAMP,
}
NO_PAREN_FUNCTIONS = {
**parser.Parser.NO_PAREN_FUNCTIONS,
TokenType.SYSTIMESTAMP: exp.Systimestamp,
}
FUNCTION_PARSERS = {
**parser.Parser.FUNCTION_PARSERS,
# https://docs.exasol.com/db/latest/sql_references/functions/alphabeticallistfunctions/listagg.htm
@ -438,6 +451,9 @@ class Exasol(Dialect):
exp.DataType.Type.DECIMAL128: "DECIMAL",
exp.DataType.Type.DECIMAL256: "DECIMAL",
exp.DataType.Type.DATETIME: "TIMESTAMP",
exp.DataType.Type.TIMESTAMPTZ: "TIMESTAMP",
exp.DataType.Type.TIMESTAMPLTZ: "TIMESTAMP",
exp.DataType.Type.TIMESTAMPNTZ: "TIMESTAMP",
}
def datatype_sql(self, expression: exp.DataType) -> str:

2
sqlglot/dialects/hive.py
View file

@ -30,6 +30,7 @@ from sqlglot.dialects.dialect import (
struct_extract_sql,
time_format,
timestrtotime_sql,
trim_sql,
unit_to_str,
var_map_sql,
sequence_sql,
@ -660,6 +661,7 @@ class Hive(Dialect):
exp.TsOrDsDiff: _date_diff_sql,
exp.TsOrDsToDate: _to_date_sql,
exp.TryCast: no_trycast_sql,
exp.Trim: trim_sql,
exp.Unicode: rename_func("ASCII"),
exp.UnixToStr: lambda self, e: self.func(
"FROM_UNIXTIME", e.this, time_format("hive")(self, e)

18
sqlglot/dialects/mysql.py
View file

@ -30,6 +30,7 @@ from sqlglot.dialects.dialect import (
from sqlglot.generator import unsupported_args
from sqlglot.helper import seq_get
from sqlglot.tokens import TokenType
from sqlglot.typing.mysql import EXPRESSION_METADATA
def _show_parser(*args: t.Any, **kwargs: t.Any) -> t.Callable[[MySQL.Parser], exp.Show]:
@ -164,6 +165,9 @@ class MySQL(Dialect):
SUPPORTS_SEMI_ANTI_JOIN = False
SAFE_DIVISION = True
SAFE_TO_ELIMINATE_DOUBLE_NEGATION = False
LEAST_GREATEST_IGNORES_NULLS = False
EXPRESSION_METADATA = EXPRESSION_METADATA.copy()
# https://prestodb.io/docs/current/functions/datetime.html#mysql-date-functions
TIME_MAPPING = {
@ -292,6 +296,7 @@ class MySQL(Dialect):
TokenType.MOD,
TokenType.SCHEMA,
TokenType.VALUES,
TokenType.CHARACTER_SET,
}
CONJUNCTION = {
@ -372,11 +377,6 @@ class MySQL(Dialect):
FUNCTION_PARSERS = {
**parser.Parser.FUNCTION_PARSERS,
"CHAR": lambda self: self.expression(
exp.Chr,
expressions=self._parse_csv(self._parse_assignment),
charset=self._match(TokenType.USING) and self._parse_var(),
),
"GROUP_CONCAT": lambda self: self._parse_group_concat(),
# https://dev.mysql.com/doc/refman/5.7/en/miscellaneous-functions.html#function_values
"VALUES": lambda self: self.expression(
@ -752,6 +752,7 @@ class MySQL(Dialect):
WRAP_DERIVED_VALUES = False
VARCHAR_REQUIRES_SIZE = True
SUPPORTS_MEDIAN = False
UPDATE_STATEMENT_SUPPORTS_FROM = False
TRANSFORMS = {
**generator.Generator.TRANSFORMS,
@ -760,6 +761,7 @@ class MySQL(Dialect):
exp.BitwiseOrAgg: rename_func("BIT_OR"),
exp.BitwiseXorAgg: rename_func("BIT_XOR"),
exp.BitwiseCount: rename_func("BIT_COUNT"),
exp.Chr: lambda self, e: self.chr_sql(e, "CHAR"),
exp.CurrentDate: no_paren_current_date_sql,
exp.DateDiff: _remove_ts_or_ds_to_date(
lambda self, e: self.func("DATEDIFF", e.this, e.expression), ("this", "expression")
@ -1301,12 +1303,6 @@ class MySQL(Dialect):
return f" LIMIT {limit_offset}"
return ""
def chr_sql(self, expression: exp.Chr) -> str:
this = self.expressions(sqls=[expression.this] + expression.expressions)
charset = expression.args.get("charset")
using = f" USING {self.sql(charset)}" if charset else ""
return f"CHAR({this}{using})"
def timestamptrunc_sql(self, expression: exp.TimestampTrunc) -> str:
unit = expression.args.get("unit")

39
sqlglot/dialects/oracle.py
View file

@ -108,6 +108,7 @@ class Oracle(Dialect):
"START": TokenType.BEGIN,
"TOP": TokenType.TOP,
"VARCHAR2": TokenType.VARCHAR,
"SYSTIMESTAMP": TokenType.SYSTIMESTAMP,
}
class Parser(parser.Parser):
@ -139,6 +140,11 @@ class Oracle(Dialect):
"DBMS_RANDOM": lambda self: self._parse_dbms_random(),
}
NO_PAREN_FUNCTIONS = {
**parser.Parser.NO_PAREN_FUNCTIONS,
TokenType.SYSTIMESTAMP: exp.Systimestamp,
}
FUNCTION_PARSERS: t.Dict[str, t.Callable] = {
**parser.Parser.FUNCTION_PARSERS,
"JSON_ARRAY": lambda self: self._parse_json_array(
@ -172,7 +178,11 @@ class Oracle(Dialect):
TYPE_LITERAL_PARSERS = {
exp.DataType.Type.DATE: lambda self, this, _: self.expression(
exp.DateStrToDate, this=this
)
),
# https://docs.oracle.com/en/database/oracle/oracle-database/19/refrn/NLS_TIMESTAMP_FORMAT.html
exp.DataType.Type.TIMESTAMP: lambda self, this, _: _build_to_timestamp(
[this, '"%Y-%m-%d %H:%M:%S.%f"']
),
}
# SELECT UNIQUE .. is old-style Oracle syntax for SELECT DISTINCT ..
@ -275,6 +285,22 @@ class Oracle(Dialect):
def _parse_connect_with_prior(self):
return self._parse_assignment()
def _parse_column_ops(self, this: t.Optional[exp.Expression]) -> t.Optional[exp.Expression]:
this = super()._parse_column_ops(this)
if not this:
return this
index = self._index
# https://docs.oracle.com/en/database/oracle/oracle-database/26/sqlrf/Interval-Expressions.html
interval_span = self._parse_interval_span(this)
if isinstance(interval_span.args.get("unit"), exp.IntervalSpan):
return interval_span
self._retreat(index)
return this
def _parse_insert_table(self) -> t.Optional[exp.Expression]:
# Oracle does not use AS for INSERT INTO alias
# https://docs.oracle.com/en/database/oracle/oracle-database/18/sqlrf/INSERT.html
@ -368,6 +394,7 @@ class Oracle(Dialect):
e: f"TO_DATE('1970-01-01', 'YYYY-MM-DD') + ({self.sql(e, 'this')} / 86400)",
exp.UtcTimestamp: rename_func("UTC_TIMESTAMP"),
exp.UtcTime: rename_func("UTC_TIME"),
exp.Systimestamp: lambda self, e: "SYSTIMESTAMP",
}
PROPERTIES_LOCATION = {
@ -420,3 +447,13 @@ class Oracle(Dialect):
def isascii_sql(self, expression: exp.IsAscii) -> str:
return f"NVL(REGEXP_LIKE({self.sql(expression.this)}, '^[' || CHR(1) || '-' || CHR(127) || ']*$'), TRUE)"
def interval_sql(self, expression: exp.Interval) -> str:
return f"{'INTERVAL ' if isinstance(expression.this, exp.Literal) else ''}{self.sql(expression, 'this')} {self.sql(expression, 'unit')}"
def columndef_sql(self, expression: exp.ColumnDef, sep: str = " ") -> str:
param_constraint = expression.find(exp.InOutColumnConstraint)
if param_constraint:
sep = f" {self.sql(param_constraint)} "
param_constraint.pop()
return super().columndef_sql(expression, sep)

48
sqlglot/dialects/postgres.py
View file

@ -13,6 +13,7 @@ from sqlglot.dialects.dialect import (
datestrtodate_sql,
build_formatted_time,
filter_array_using_unnest,
getbit_sql,
inline_array_sql,
json_extract_segments,
json_path_key_only_name,
@ -358,8 +359,6 @@ class Postgres(Dialect):
"<@": TokenType.LT_AT,
"?&": TokenType.QMARK_AMP,
"?|": TokenType.QMARK_PIPE,
"&<": TokenType.AMP_LT,
"&>": TokenType.AMP_GT,
"#-": TokenType.HASH_DASH,
"|/": TokenType.PIPE_SLASH,
"||/": TokenType.DPIPE_SLASH,
@ -376,7 +375,7 @@ class Postgres(Dialect):
"NAME": TokenType.NAME,
"OID": TokenType.OBJECT_IDENTIFIER,
"ONLY": TokenType.ONLY,
"OPERATOR": TokenType.OPERATOR,
"POINT": TokenType.POINT,
"REFRESH": TokenType.COMMAND,
"REINDEX": TokenType.COMMAND,
"RESET": TokenType.COMMAND,
@ -424,6 +423,9 @@ class Postgres(Dialect):
FUNCTIONS = {
**parser.Parser.FUNCTIONS,
"ARRAY_PREPEND": lambda args: exp.ArrayPrepend(
this=seq_get(args, 1), expression=seq_get(args, 0)
),
"BIT_AND": exp.BitwiseAndAgg.from_arg_list,
"BIT_OR": exp.BitwiseOrAgg.from_arg_list,
"BIT_XOR": exp.BitwiseXorAgg.from_arg_list,
@ -432,6 +434,9 @@ class Postgres(Dialect):
binary_from_function(exp.IntDiv)(args), exp.DataType.Type.DECIMAL
),
"GENERATE_SERIES": _build_generate_series,
"GET_BIT": lambda args: exp.Getbit(
this=seq_get(args, 0), expression=seq_get(args, 1), zero_is_msb=True
),
"JSON_EXTRACT_PATH": build_json_extract_path(exp.JSONExtract),
"JSON_EXTRACT_PATH_TEXT": build_json_extract_path(exp.JSONExtractScalar),
"LENGTH": lambda args: exp.Length(this=seq_get(args, 0), encoding=seq_get(args, 1)),
@ -449,6 +454,11 @@ class Postgres(Dialect):
"LEVENSHTEIN_LESS_EQUAL": _build_levenshtein_less_equal,
"JSON_OBJECT_AGG": lambda args: exp.JSONObjectAgg(expressions=args),
"JSONB_OBJECT_AGG": exp.JSONBObjectAgg.from_arg_list,
"WIDTH_BUCKET": lambda args: exp.WidthBucket(
this=seq_get(args, 0), threshold=seq_get(args, 1)
)
if len(args) == 2
else exp.WidthBucket.from_arg_list(args),
}
NO_PAREN_FUNCTIONS = {
@ -479,12 +489,9 @@ class Postgres(Dialect):
RANGE_PARSERS = {
**parser.Parser.RANGE_PARSERS,
TokenType.DAMP: binary_range_parser(exp.ArrayOverlaps),
TokenType.AMP_LT: binary_range_parser(exp.ExtendsLeft),
TokenType.AMP_GT: binary_range_parser(exp.ExtendsRight),
TokenType.DAT: lambda self, this: self.expression(
exp.MatchAgainst, this=self._parse_bitwise(), expressions=[this]
),
TokenType.OPERATOR: lambda self, this: self._parse_operator(this),
}
STATEMENT_PARSERS = {
@ -517,29 +524,6 @@ class Postgres(Dialect):
self._match_text_seq("S")
return self.expression(exp.Placeholder, this=this)
def _parse_operator(self, this: t.Optional[exp.Expression]) -> t.Optional[exp.Expression]:
while True:
if not self._match(TokenType.L_PAREN):
break
op = ""
while self._curr and not self._match(TokenType.R_PAREN):
op += self._curr.text
self._advance()
this = self.expression(
exp.Operator,
comments=self._prev_comments,
this=this,
operator=op,
expression=self._parse_bitwise(),
)
if not self._match(TokenType.OPERATOR):
break
return this
def _parse_date_part(self) -> exp.Expression:
part = self._parse_type()
self._match(TokenType.COMMA)
@ -636,6 +620,7 @@ class Postgres(Dialect):
exp.AnyValue: _versioned_anyvalue_sql,
exp.ArrayConcat: lambda self, e: self.arrayconcat_sql(e, name="ARRAY_CAT"),
exp.ArrayFilter: filter_array_using_unnest,
exp.ArrayPrepend: lambda self, e: self.func("ARRAY_PREPEND", e.expression, e.this),
exp.BitwiseAndAgg: rename_func("BIT_AND"),
exp.BitwiseOrAgg: rename_func("BIT_OR"),
exp.BitwiseXor: lambda self, e: self.binary(e, "#"),
@ -650,6 +635,7 @@ class Postgres(Dialect):
exp.DateSub: _date_add_sql("-"),
exp.Explode: rename_func("UNNEST"),
exp.ExplodingGenerateSeries: rename_func("GENERATE_SERIES"),
exp.Getbit: getbit_sql,
exp.GroupConcat: lambda self, e: groupconcat_sql(
self, e, func_name="STRING_AGG", within_group=False
),
@ -714,7 +700,9 @@ class Postgres(Dialect):
exp.TimestampTrunc: timestamptrunc_sql(zone=True),
exp.TimeStrToTime: timestrtotime_sql,
exp.TimeToStr: lambda self, e: self.func("TO_CHAR", e.this, self.format_time(e)),
exp.ToChar: lambda self, e: self.function_fallback_sql(e),
exp.ToChar: lambda self, e: self.function_fallback_sql(e)
if e.args.get("format")
else self.tochar_sql(e),
exp.Trim: trim_sql,
exp.TryCast: no_trycast_sql,
exp.TsOrDsAdd: _date_add_sql("+"),

2
sqlglot/dialects/presto.py
View file

@ -267,6 +267,7 @@ class Presto(Dialect):
TABLESAMPLE_SIZE_IS_PERCENT = True
LOG_BASE_FIRST: t.Optional[bool] = None
SUPPORTS_VALUES_DEFAULT = False
LEAST_GREATEST_IGNORES_NULLS = False
TIME_MAPPING = MySQL.TIME_MAPPING
@ -373,6 +374,7 @@ class Presto(Dialect):
"MD5": exp.MD5Digest.from_arg_list,
"SHA256": lambda args: exp.SHA2(this=seq_get(args, 0), length=exp.Literal.number(256)),
"SHA512": lambda args: exp.SHA2(this=seq_get(args, 0), length=exp.Literal.number(512)),
"WEEK": exp.WeekOfYear.from_arg_list,
}
FUNCTION_PARSERS = parser.Parser.FUNCTION_PARSERS.copy()

4
sqlglot/dialects/redshift.py
View file

@ -82,6 +82,7 @@ class Redshift(Postgres):
),
"STRTOL": exp.FromBase.from_arg_list,
}
FUNCTIONS.pop("GET_BIT")
NO_PAREN_FUNCTION_PARSERS = {
**Postgres.Parser.NO_PAREN_FUNCTION_PARSERS,
@ -244,6 +245,9 @@ class Redshift(Postgres):
TRANSFORMS.pop(exp.LastDay)
TRANSFORMS.pop(exp.SHA2)
# Postgres and Redshift have different semantics for Getbit
TRANSFORMS.pop(exp.Getbit)
# Postgres does not permit a double precision argument in ROUND; Redshift does
TRANSFORMS.pop(exp.Round)

16
sqlglot/dialects/singlestore.py
View file

@ -82,6 +82,7 @@ class SingleStore(MySQL):
"::$": TokenType.DCOLONDOLLAR,
"::%": TokenType.DCOLONPERCENT,
"::?": TokenType.DCOLONQMARK,
"RECORD": TokenType.STRUCT,
}
class Parser(MySQL.Parser):
@ -176,6 +177,10 @@ class SingleStore(MySQL):
expression=seq_get(args, 0),
json_type="JSON",
),
"JSON_KEYS": lambda args: exp.JSONKeys(
this=seq_get(args, 0),
expressions=args[1:],
),
"JSON_PRETTY": exp.JSONFormat.from_arg_list,
"JSON_BUILD_ARRAY": lambda args: exp.JSONArray(expressions=args),
"JSON_BUILD_OBJECT": lambda args: exp.JSONObject(expressions=args),
@ -328,6 +333,7 @@ class SingleStore(MySQL):
SUPPORTS_UESCAPE = False
NULL_ORDERING_SUPPORTED = True
MATCH_AGAINST_TABLE_PREFIX = "TABLE "
STRUCT_DELIMITER = ("(", ")")
@staticmethod
def _unicode_substitute(m: re.Match[str]) -> str:
@ -497,7 +503,6 @@ class SingleStore(MySQL):
),
exp.IsAscii: lambda self, e: f"({self.sql(e, 'this')} RLIKE '^[\x00-\x7f]*$')",
exp.MD5Digest: lambda self, e: self.func("UNHEX", self.func("MD5", e.this)),
exp.Chr: rename_func("CHAR"),
exp.Contains: rename_func("INSTR"),
exp.RegexpExtractAll: unsupported_args("position", "occurrence", "group")(
lambda self, e: self.func(
@ -613,7 +618,6 @@ class SingleStore(MySQL):
exp.DataType.Type.SERIAL,
exp.DataType.Type.SMALLSERIAL,
exp.DataType.Type.SMALLMONEY,
exp.DataType.Type.STRUCT,
exp.DataType.Type.SUPER,
exp.DataType.Type.TIMETZ,
exp.DataType.Type.TIMESTAMPNTZ,
@ -654,6 +658,7 @@ class SingleStore(MySQL):
exp.DataType.Type.LINESTRING: "GEOGRAPHY",
exp.DataType.Type.POLYGON: "GEOGRAPHY",
exp.DataType.Type.MULTIPOLYGON: "GEOGRAPHY",
exp.DataType.Type.STRUCT: "RECORD",
exp.DataType.Type.JSONB: "BSON",
exp.DataType.Type.TIMESTAMP: "TIMESTAMP",
exp.DataType.Type.TIMESTAMP_S: "TIMESTAMP",
@ -1760,8 +1765,13 @@ class SingleStore(MySQL):
self.func("TO_JSON", expression.this),
)
@unsupported_args("kind", "nested", "values")
@unsupported_args("kind", "values")
def datatype_sql(self, expression: exp.DataType) -> str:
if expression.args.get("nested") and not expression.is_type(exp.DataType.Type.STRUCT):
self.unsupported(
f"Argument 'nested' is not supported for representation of '{expression.this.value}' in SingleStore"
)
if expression.is_type(exp.DataType.Type.VARBINARY) and not expression.expressions:
# `VARBINARY` must always have a size - if it doesn't, we always generate `BLOB`
return "BLOB"

146
sqlglot/dialects/snowflake.py
View file

@ -32,7 +32,7 @@ from sqlglot.dialects.dialect import (
groupconcat_sql,
)
from sqlglot.generator import unsupported_args
from sqlglot.helper import find_new_name, flatten, is_float, is_int, seq_get
from sqlglot.helper import find_new_name, flatten, is_date_unit, is_int, seq_get
from sqlglot.optimizer.scope import build_scope, find_all_in_scope
from sqlglot.tokens import TokenType
from sqlglot.typing.snowflake import EXPRESSION_METADATA
@ -77,6 +77,15 @@ def _build_approx_top_k(args: t.List) -> exp.ApproxTopK:
return exp.ApproxTopK.from_arg_list(args)
def _build_date_from_parts(args: t.List) -> exp.DateFromParts:
return exp.DateFromParts(
year=seq_get(args, 0),
month=seq_get(args, 1),
day=seq_get(args, 2),
allow_overflow=True,
)
def _build_datetime(
name: str, kind: exp.DataType.Type, safe: bool = False
) -> t.Callable[[t.List], exp.Func]:
@ -87,7 +96,7 @@ def _build_datetime(
int_value = value is not None and is_int(value.name)
int_scale_or_fmt = scale_or_fmt is not None and scale_or_fmt.is_int
if isinstance(value, exp.Literal) or (value and scale_or_fmt):
if isinstance(value, (exp.Literal, exp.Neg)) or (value and scale_or_fmt):
# Converts calls like `TO_TIME('01:02:03')` into casts
if len(args) == 1 and value.is_string and not int_value:
return (
@ -99,17 +108,26 @@ def _build_datetime(
# Handles `TO_TIMESTAMP(str, fmt)` and `TO_TIMESTAMP(num, scale)` as special
# cases so we can transpile them, since they're relatively common
if kind in TIMESTAMP_TYPES:
if not safe and (int_value or int_scale_or_fmt):
if not safe and (int_scale_or_fmt or (int_value and scale_or_fmt is None)):
# TRY_TO_TIMESTAMP('integer') is not parsed into exp.UnixToTime as
# it's not easily transpilable
return exp.UnixToTime(this=value, scale=scale_or_fmt)
if not int_scale_or_fmt and not is_float(value.name):
expr = build_formatted_time(exp.StrToTime, "snowflake")(args)
expr.set("safe", safe)
expr.set("target_type", exp.DataType.build(kind, dialect="snowflake"))
return expr
# it's not easily transpilable. Also, numeric-looking strings with
# format strings (e.g., TO_TIMESTAMP('20240115', 'YYYYMMDD')) should
# use StrToTime, not UnixToTime.
unix_expr = exp.UnixToTime(this=value, scale=scale_or_fmt)
unix_expr.set("target_type", exp.DataType.build(kind, dialect="snowflake"))
return unix_expr
if scale_or_fmt and not int_scale_or_fmt:
# Format string provided (e.g., 'YYYY-MM-DD'), use StrToTime
strtotime_expr = build_formatted_time(exp.StrToTime, "snowflake")(args)
strtotime_expr.set("safe", safe)
strtotime_expr.set("target_type", exp.DataType.build(kind, dialect="snowflake"))
return strtotime_expr
if kind in (exp.DataType.Type.DATE, exp.DataType.Type.TIME) and not int_value:
# Handle DATE/TIME with format strings - allow int_value if a format string is provided
has_format_string = scale_or_fmt and not int_scale_or_fmt
if kind in (exp.DataType.Type.DATE, exp.DataType.Type.TIME) and (
not int_value or has_format_string
):
klass = exp.TsOrDsToDate if kind == exp.DataType.Type.DATE else exp.TsOrDsToTime
formatted_exp = build_formatted_time(klass, "snowflake")(args)
formatted_exp.set("safe", safe)
@ -135,7 +153,10 @@ def _build_object_construct(args: t.List) -> t.Union[exp.StarMap, exp.Struct]:
def _build_datediff(args: t.List) -> exp.DateDiff:
return exp.DateDiff(
this=seq_get(args, 2), expression=seq_get(args, 1), unit=map_date_part(seq_get(args, 0))
this=seq_get(args, 2),
expression=seq_get(args, 1),
unit=map_date_part(seq_get(args, 0)),
date_part_boundary=True,
)
@ -160,7 +181,13 @@ def _build_bitwise(expr_type: t.Type[B], name: str) -> t.Callable[[t.List], B |
)
return exp.Anonymous(this=name, expressions=args)
return binary_from_function(expr_type)(args)
result = binary_from_function(expr_type)(args)
# Snowflake specifies INT128 for bitwise shifts
if expr_type in (exp.BitwiseLeftShift, exp.BitwiseRightShift):
result.set("requires_int128", True)
return result
return _builder
@ -242,7 +269,13 @@ def _show_parser(*args: t.Any, **kwargs: t.Any) -> t.Callable[[Snowflake.Parser]
def _date_trunc_to_time(args: t.List) -> exp.DateTrunc | exp.TimestampTrunc:
trunc = date_trunc_to_time(args)
trunc.set("unit", map_date_part(trunc.args["unit"]))
unit = map_date_part(trunc.args["unit"])
trunc.set("unit", unit)
is_time_input = trunc.this.is_type(exp.DataType.Type.TIME, exp.DataType.Type.TIMETZ)
if (isinstance(trunc, exp.TimestampTrunc) and is_date_unit(unit) or is_time_input) or (
isinstance(trunc, exp.DateTrunc) and not is_date_unit(unit)
):
trunc.set("input_type_preserved", True)
return trunc
@ -629,6 +662,7 @@ class Snowflake(Dialect):
ALTER_TABLE_ADD_REQUIRED_FOR_EACH_COLUMN = False
TRY_CAST_REQUIRES_STRING = True
SUPPORTS_ALIAS_REFS_IN_JOIN_CONDITIONS = True
LEAST_GREATEST_IGNORES_NULLS = False
EXPRESSION_METADATA = EXPRESSION_METADATA.copy()
@ -660,6 +694,12 @@ class Snowflake(Dialect):
"ss": "%S",
"FF6": "%f",
"ff6": "%f",
# Seems like Snowflake treats AM/PM in the format string as equivalent,
# only the time (stamp) value's AM/PM affects the output
"AM": "%p",
"am": "%p",
"PM": "%p",
"pm": "%p",
}
DATE_PART_MAPPING = {
@ -701,6 +741,11 @@ class Snowflake(Dialect):
FUNCTIONS = {
**parser.Parser.FUNCTIONS,
"ADD_MONTHS": lambda args: exp.AddMonths(
this=seq_get(args, 0),
expression=seq_get(args, 1),
preserve_end_of_month=True,
),
"APPROX_PERCENTILE": exp.ApproxQuantile.from_arg_list,
"APPROX_TOP_K": _build_approx_top_k,
"ARRAY_CONSTRUCT": lambda args: exp.Array(expressions=args),
@ -740,10 +785,18 @@ class Snowflake(Dialect):
"BIT_XORAGG": exp.BitwiseXorAgg.from_arg_list,
"BITMAP_OR_AGG": exp.BitmapOrAgg.from_arg_list,
"BOOLXOR": _build_bitwise(exp.Xor, "BOOLXOR"),
"CORR": lambda args: exp.Corr(
this=seq_get(args, 0),
expression=seq_get(args, 1),
null_on_zero_variance=True,
),
"DATE": _build_datetime("DATE", exp.DataType.Type.DATE),
"DATEFROMPARTS": _build_date_from_parts,
"DATE_FROM_PARTS": _build_date_from_parts,
"DATE_TRUNC": _date_trunc_to_time,
"DATEADD": _build_date_time_add(exp.DateAdd),
"DATEDIFF": _build_datediff,
"DAYNAME": lambda args: exp.Dayname(this=seq_get(args, 0), abbreviated=True),
"DAYOFWEEKISO": exp.DayOfWeekIso.from_arg_list,
"DIV0": _build_if_from_div0,
"DIV0NULL": _build_if_from_div0null,
@ -758,19 +811,28 @@ class Snowflake(Dialect):
expression=dialect.to_json_path(seq_get(args, 1)),
requires_json=True,
),
"GREATEST_IGNORE_NULLS": lambda args: exp.Greatest(
this=seq_get(args, 0), expressions=args[1:], ignore_nulls=True
),
"LEAST_IGNORE_NULLS": lambda args: exp.Least(
this=seq_get(args, 0), expressions=args[1:], ignore_nulls=True
),
"HEX_DECODE_BINARY": exp.Unhex.from_arg_list,
"IFF": exp.If.from_arg_list,
"MD5_HEX": exp.MD5.from_arg_list,
"MD5_BINARY": exp.MD5Digest.from_arg_list,
"MD5_NUMBER_LOWER64": exp.MD5NumberLower64.from_arg_list,
"MD5_NUMBER_UPPER64": exp.MD5NumberUpper64.from_arg_list,
"MONTHNAME": lambda args: exp.Monthname(this=seq_get(args, 0), abbreviated=True),
"LAST_DAY": lambda args: exp.LastDay(
this=seq_get(args, 0), unit=map_date_part(seq_get(args, 1))
),
"LEN": lambda args: exp.Length(this=seq_get(args, 0), binary=True),
"LENGTH": lambda args: exp.Length(this=seq_get(args, 0), binary=True),
"LOCALTIMESTAMP": exp.CurrentTimestamp.from_arg_list,
"NULLIFZERO": _build_if_from_nullifzero,
"OBJECT_CONSTRUCT": _build_object_construct,
"OBJECT_KEYS": exp.JSONKeys.from_arg_list,
"OCTET_LENGTH": exp.ByteLength.from_arg_list,
"PARSE_URL": lambda args: exp.ParseUrl(
this=seq_get(args, 0), permissive=seq_get(args, 1)
@ -787,6 +849,7 @@ class Snowflake(Dialect):
"SHA2_BINARY": exp.SHA2Digest.from_arg_list,
"SHA2_HEX": exp.SHA2.from_arg_list,
"SQUARE": lambda args: exp.Pow(this=seq_get(args, 0), expression=exp.Literal.number(2)),
"STDDEV_SAMP": exp.Stddev.from_arg_list,
"STRTOK": _build_strtok,
"SYSDATE": lambda args: exp.CurrentTimestamp(this=seq_get(args, 0), sysdate=True),
"TABLE": lambda args: exp.TableFromRows(this=seq_get(args, 0)),
@ -798,6 +861,22 @@ class Snowflake(Dialect):
"TIMESTAMP_FROM_PARTS": _build_timestamp_from_parts,
"TIMESTAMPNTZFROMPARTS": _build_timestamp_from_parts,
"TIMESTAMP_NTZ_FROM_PARTS": _build_timestamp_from_parts,
"TRY_DECRYPT": lambda args: exp.Decrypt(
this=seq_get(args, 0),
passphrase=seq_get(args, 1),
aad=seq_get(args, 2),
encryption_method=seq_get(args, 3),
safe=True,
),
"TRY_DECRYPT_RAW": lambda args: exp.DecryptRaw(
this=seq_get(args, 0),
key=seq_get(args, 1),
iv=seq_get(args, 2),
aad=seq_get(args, 3),
encryption_method=seq_get(args, 4),
aead=seq_get(args, 5),
safe=True,
),
"TRY_PARSE_JSON": lambda args: exp.ParseJSON(this=seq_get(args, 0), safe=True),
"TRY_TO_BINARY": lambda args: exp.ToBinary(
this=seq_get(args, 0), format=seq_get(args, 1), safe=True
@ -817,6 +896,15 @@ class Snowflake(Dialect):
"TRY_TO_TIMESTAMP": _build_datetime(
"TRY_TO_TIMESTAMP", exp.DataType.Type.TIMESTAMP, safe=True
),
"TRY_TO_TIMESTAMP_LTZ": _build_datetime(
"TRY_TO_TIMESTAMP_LTZ", exp.DataType.Type.TIMESTAMPLTZ, safe=True
),
"TRY_TO_TIMESTAMP_NTZ": _build_datetime(
"TRY_TO_TIMESTAMP_NTZ", exp.DataType.Type.TIMESTAMPNTZ, safe=True
),
"TRY_TO_TIMESTAMP_TZ": _build_datetime(
"TRY_TO_TIMESTAMP_TZ", exp.DataType.Type.TIMESTAMPTZ, safe=True
),
"TO_CHAR": build_timetostr_or_tochar,
"TO_DATE": _build_datetime("TO_DATE", exp.DataType.Type.DATE),
**dict.fromkeys(
@ -1430,11 +1518,30 @@ class Snowflake(Dialect):
exp.CurrentTimestamp: lambda self, e: self.func("SYSDATE")
if e.args.get("sysdate")
else self.function_fallback_sql(e),
exp.Localtimestamp: lambda self, e: self.func("CURRENT_TIMESTAMP", e.this)
if e.this
else "CURRENT_TIMESTAMP",
exp.DateAdd: date_delta_sql("DATEADD"),
exp.DateDiff: date_delta_sql("DATEDIFF"),
exp.DatetimeAdd: date_delta_sql("TIMESTAMPADD"),
exp.DatetimeDiff: timestampdiff_sql,
exp.DateStrToDate: datestrtodate_sql,
exp.Decrypt: lambda self, e: self.func(
f"{'TRY_' if e.args.get('safe') else ''}DECRYPT",
e.this,
e.args.get("passphrase"),
e.args.get("aad"),
e.args.get("encryption_method"),
),
exp.DecryptRaw: lambda self, e: self.func(
f"{'TRY_' if e.args.get('safe') else ''}DECRYPT_RAW",
e.this,
e.args.get("key"),
e.args.get("iv"),
e.args.get("aad"),
e.args.get("encryption_method"),
e.args.get("aead"),
),
exp.DayOfMonth: rename_func("DAYOFMONTH"),
exp.DayOfWeek: rename_func("DAYOFWEEK"),
exp.DayOfWeekIso: rename_func("DAYOFWEEKISO"),
@ -1461,6 +1568,7 @@ class Snowflake(Dialect):
exp.JSONExtractScalar: lambda self, e: self.func(
"JSON_EXTRACT_PATH_TEXT", e.this, e.expression
),
exp.JSONKeys: rename_func("OBJECT_KEYS"),
exp.JSONObject: lambda self, e: self.func("OBJECT_CONSTRUCT_KEEP_NULL", *e.expressions),
exp.JSONPathRoot: lambda *_: "",
exp.JSONValueArray: _json_extract_value_array_sql,
@ -1566,7 +1674,7 @@ class Snowflake(Dialect):
f"{'TRY_' if e.args.get('safe') else ''}TO_TIME", e.this, self.format_time(e)
),
exp.Unhex: rename_func("HEX_DECODE_BINARY"),
exp.UnixToTime: rename_func("TO_TIMESTAMP"),
exp.UnixToTime: lambda self, e: self.func("TO_TIMESTAMP", e.this, e.args.get("scale")),
exp.Uuid: rename_func("UUID_STRING"),
exp.VarMap: lambda self, e: var_map_sql(self, e, "OBJECT_CONSTRUCT"),
exp.Booland: rename_func("BOOLAND"),
@ -1707,6 +1815,14 @@ class Snowflake(Dialect):
return super().log_sql(expression)
def greatest_sql(self, expression: exp.Greatest) -> str:
name = "GREATEST_IGNORE_NULLS" if expression.args.get("ignore_nulls") else "GREATEST"
return self.func(name, expression.this, *expression.expressions)
def least_sql(self, expression: exp.Least) -> str:
name = "LEAST_IGNORE_NULLS" if expression.args.get("ignore_nulls") else "LEAST"
return self.func(name, expression.this, *expression.expressions)
def unnest_sql(self, expression: exp.Unnest) -> str:
unnest_alias = expression.args.get("alias")
offset = expression.args.get("offset")

10
sqlglot/dialects/spark.py
View file

@ -14,6 +14,7 @@ from sqlglot.dialects.dialect import (
)
from sqlglot.dialects.hive import _build_with_ignore_nulls
from sqlglot.dialects.spark2 import Spark2, temporary_storage_provider, _build_as_cast
from sqlglot.typing.spark import EXPRESSION_METADATA
from sqlglot.helper import ensure_list, seq_get
from sqlglot.tokens import TokenType
from sqlglot.transforms import (
@ -112,6 +113,7 @@ def _groupconcat_sql(self: Spark.Generator, expression: exp.GroupConcat) -> str:
class Spark(Spark2):
SUPPORTS_ORDER_BY_ALL = True
SUPPORTS_NULL_TYPE = True
EXPRESSION_METADATA = EXPRESSION_METADATA.copy()
class Tokenizer(Spark2.Tokenizer):
STRING_ESCAPES_ALLOWED_IN_RAW_STRINGS = False
@ -139,6 +141,7 @@ class Spark(Spark2):
"TRY_SUBTRACT": exp.SafeSubtract.from_arg_list,
"DATEDIFF": _build_datediff,
"DATE_DIFF": _build_datediff,
"JSON_OBJECT_KEYS": exp.JSONKeys.from_arg_list,
"LISTAGG": exp.GroupConcat.from_arg_list,
"TIMESTAMP_LTZ": _build_as_cast("TIMESTAMP_LTZ"),
"TIMESTAMP_NTZ": _build_as_cast("TIMESTAMP_NTZ"),
@ -174,6 +177,12 @@ class Spark(Spark2):
return self.expression(exp.ComputedColumnConstraint, this=this.expression)
return this
def _parse_pivot_aggregation(self) -> t.Optional[exp.Expression]:
# Spark 3+ and Databricks support non aggregate functions in PIVOT too, e.g
# PIVOT (..., 'foo' AS bar FOR col_to_pivot IN (...))
aggregate_expr = self._parse_function() or self._parse_disjunction()
return self._parse_alias(aggregate_expr)
class Generator(Spark2.Generator):
SUPPORTS_TO_NUMBER = True
PAD_FILL_PATTERN_IS_REQUIRED = False
@ -214,6 +223,7 @@ class Spark(Spark2):
exp.DatetimeSub: date_delta_to_binary_interval_op(cast=False),
exp.GroupConcat: _groupconcat_sql,
exp.EndsWith: rename_func("ENDSWITH"),
exp.JSONKeys: rename_func("JSON_OBJECT_KEYS"),
exp.PartitionedByProperty: lambda self,
e: f"PARTITIONED BY {self.wrap(self.expressions(sqls=[_normalize_partition(e) for e in e.this.expressions], skip_first=True))}",
exp.SafeAdd: rename_func("TRY_ADD"),

5
sqlglot/dialects/spark2.py
View file

@ -9,11 +9,11 @@ from sqlglot.dialects.dialect import (
is_parse_json,
pivot_column_names,
rename_func,
trim_sql,
unit_to_str,
)
from sqlglot.dialects.hive import Hive
from sqlglot.helper import seq_get
from sqlglot.parser import build_trim
from sqlglot.tokens import TokenType
from sqlglot.transforms import (
preprocess,
@ -159,9 +159,11 @@ class Spark2(Hive):
),
zone=seq_get(args, 1),
),
"LTRIM": lambda args: build_trim(args, reverse_args=True),
"INT": _build_as_cast("int"),
"MAP_FROM_ARRAYS": exp.Map.from_arg_list,
"RLIKE": exp.RegexpLike.from_arg_list,
"RTRIM": lambda args: build_trim(args, is_left=False, reverse_args=True),
"SHIFTLEFT": binary_from_function(exp.BitwiseLeftShift),
"SHIFTRIGHT": binary_from_function(exp.BitwiseRightShift),
"STRING": _build_as_cast("string"),
@ -288,7 +290,6 @@ class Spark2(Hive):
exp.StrToDate: _str_to_date,
exp.StrToTime: lambda self, e: self.func("TO_TIMESTAMP", e.this, self.format_time(e)),
exp.TimestampTrunc: lambda self, e: self.func("DATE_TRUNC", unit_to_str(e), e.this),
exp.Trim: trim_sql,
exp.UnixToTime: _unix_to_time_sql,
exp.VariancePop: rename_func("VAR_POP"),
exp.WeekOfYear: rename_func("WEEKOFYEAR"),

1
sqlglot/dialects/starrocks.py
View file

@ -126,6 +126,7 @@ class StarRocks(MySQL):
VARCHAR_REQUIRES_SIZE = False
PARSE_JSON_NAME: t.Optional[str] = "PARSE_JSON"
WITH_PROPERTIES_PREFIX = "PROPERTIES"
UPDATE_STATEMENT_SUPPORTS_FROM = True
# StarRocks doesn't support "IS TRUE/FALSE" syntax.
IS_BOOL_ALLOWED = False

1
sqlglot/dialects/trino.py
View file

@ -69,6 +69,7 @@ class Trino(Presto):
)
class Generator(Presto.Generator):
EXCEPT_INTERSECT_SUPPORT_ALL_CLAUSE = True
PROPERTIES_LOCATION = {
**Presto.Generator.PROPERTIES_LOCATION,
exp.LocationProperty: exp.Properties.Location.POST_WITH,

3
sqlglot/dialects/tsql.py
View file

@ -989,6 +989,9 @@ class TSQL(Dialect):
return expression
def _parse_primary_key_part(self) -> t.Optional[exp.Expression]:
return self._parse_ordered()
class Generator(generator.Generator):
LIMIT_IS_TOP = True
QUERY_HINTS = False

2
sqlglot/diff.py
View file

@ -181,7 +181,7 @@ class ChangeDistiller:
def __init__(self, f: float = 0.6, t: float = 0.6, dialect: DialectType = None) -> None:
self.f = f
self.t = t
self._sql_generator = Dialect.get_or_raise(dialect).generator()
self._sql_generator = Dialect.get_or_raise(dialect).generator(comments=False)
def diff(
self,

166
sqlglot/expressions.py
View file

@ -2134,6 +2134,11 @@ class ComputedColumnConstraint(ColumnConstraintKind):
arg_types = {"this": True, "persisted": False, "not_null": False, "data_type": False}
# https://docs.oracle.com/en/database/other-databases/timesten/22.1/plsql-developer/examples-using-input-and-output-parameters-and-bind-variables.html#GUID-4B20426E-F93F-4835-88CB-6A79829A8D7F
class InOutColumnConstraint(ColumnConstraintKind):
arg_types = {"input_": False, "output": False}
class Constraint(Expression):
arg_types = {"this": True, "expressions": True}
@ -2481,6 +2486,7 @@ class OnConflict(Expression):
"expressions": False,
"action": False,
"conflict_keys": False,
"index_predicate": False,
"constraint": False,
"where": False,
}
@ -5143,7 +5149,7 @@ class BitwiseAnd(Binary):
class BitwiseLeftShift(Binary):
pass
arg_types = {"this": True, "expression": True, "requires_int128": False}
class BitwiseOr(Binary):
@ -5151,7 +5157,7 @@ class BitwiseOr(Binary):
class BitwiseRightShift(Binary):
pass
arg_types = {"this": True, "expression": True, "requires_int128": False}
class BitwiseXor(Binary):
@ -5313,6 +5319,12 @@ class Sub(Binary):
pass
# https://www.postgresql.org/docs/current/functions-range.html
# Represents range adjacency operator: -|-
class Adjacent(Binary):
pass
# Unary Expressions
# (NOT a)
class Unary(Condition):
@ -6041,6 +6053,14 @@ class ArrayAny(Func):
arg_types = {"this": True, "expression": True}
class ArrayAppend(Func):
arg_types = {"this": True, "expression": True}
class ArrayPrepend(Func):
arg_types = {"this": True, "expression": True}
class ArrayConcat(Func):
_sql_names = ["ARRAY_CONCAT", "ARRAY_CAT"]
arg_types = {"this": True, "expressions": False}
@ -6410,6 +6430,10 @@ class Localtimestamp(Func):
arg_types = {"this": False}
class Systimestamp(Func):
arg_types = {"this": False}
class CurrentTimestamp(Func):
arg_types = {"this": False, "sysdate": False}
@ -6495,7 +6519,7 @@ class DateDiff(Func, TimeUnit):
class DateTrunc(Func):
arg_types = {"unit": True, "this": True, "zone": False}
arg_types = {"unit": True, "this": True, "zone": False, "input_type_preserved": False}
def __init__(self, **args):
# Across most dialects it's safe to unabbreviate the unit (e.g. 'Q' -> 'QUARTER') except Oracle
@ -6563,6 +6587,10 @@ class DayOfYear(Func):
_sql_names = ["DAY_OF_YEAR", "DAYOFYEAR"]
class Dayname(Func):
arg_types = {"this": True, "abbreviated": False}
class ToDays(Func):
pass
@ -6587,6 +6615,7 @@ class MakeInterval(Func):
arg_types = {
"year": False,
"month": False,
"week": False,
"day": False,
"hour": False,
"minute": False,
@ -6650,7 +6679,7 @@ class TimestampDiff(Func, TimeUnit):
class TimestampTrunc(Func, TimeUnit):
arg_types = {"this": True, "unit": True, "zone": False}
arg_types = {"this": True, "unit": True, "zone": False, "input_type_preserved": False}
class TimeSlice(Func, TimeUnit):
@ -6675,7 +6704,7 @@ class TimeTrunc(Func, TimeUnit):
class DateFromParts(Func):
_sql_names = ["DATE_FROM_PARTS", "DATEFROMPARTS"]
arg_types = {"year": True, "month": False, "day": False}
arg_types = {"year": True, "month": False, "day": False, "allow_overflow": False}
class TimeFromParts(Func):
@ -6721,6 +6750,30 @@ class DecodeCase(Func):
is_var_len_args = True
# https://docs.snowflake.com/en/sql-reference/functions/decrypt
class Decrypt(Func):
arg_types = {
"this": True,
"passphrase": True,
"aad": False,
"encryption_method": False,
"safe": False,
}
# https://docs.snowflake.com/en/sql-reference/functions/decrypt_raw
class DecryptRaw(Func):
arg_types = {
"this": True,
"key": True,
"iv": True,
"aad": False,
"encryption_method": False,
"aead": False,
"safe": False,
}
class DenseRank(AggFunc):
arg_types = {"expressions": False}
is_var_len_args = True
@ -6734,6 +6787,16 @@ class Encode(Func):
arg_types = {"this": True, "charset": True}
# https://docs.snowflake.com/en/sql-reference/functions/encrypt
class Encrypt(Func):
arg_types = {"this": True, "passphrase": True, "aad": False, "encryption_method": False}
# https://docs.snowflake.com/en/sql-reference/functions/encrypt_raw
class EncryptRaw(Func):
arg_types = {"this": True, "key": True, "iv": True, "aad": False, "encryption_method": False}
class EqualNull(Func):
arg_types = {"this": True, "expression": True}
@ -6882,21 +6945,13 @@ class GetExtract(Func):
class Getbit(Func):
arg_types = {"this": True, "expression": True}
_sql_names = ["GETBIT", "GET_BIT"]
# zero_is_msb means the most significant bit is indexed 0
arg_types = {"this": True, "expression": True, "zero_is_msb": False}
class Greatest(Func):
arg_types = {"this": True, "expressions": False, "null_if_any_null": False}
is_var_len_args = True
class GreatestIgnoreNulls(Func):
arg_types = {"expressions": True}
is_var_len_args = True
class LeastIgnoreNulls(Func):
arg_types = {"expressions": True}
arg_types = {"this": True, "expressions": False, "ignore_nulls": True}
is_var_len_args = True
@ -6966,6 +7021,7 @@ class Or(Connector, Func):
class Xor(Connector, Func):
arg_types = {"this": False, "expression": False, "expressions": False}
is_var_len_args = True
class If(Func):
@ -7069,6 +7125,12 @@ class Format(Func):
is_var_len_args = True
class JSONKeys(Func):
arg_types = {"this": True, "expression": False, "expressions": False}
is_var_len_args = True
_sql_names = ["JSON_KEYS"]
class JSONKeyValue(Expression):
arg_types = {"this": True, "expression": True}
@ -7364,7 +7426,7 @@ class ParseDatetime(Func):
class Least(Func):
arg_types = {"this": True, "expressions": False, "null_if_any_null": False}
arg_types = {"this": True, "expressions": False, "ignore_nulls": True}
is_var_len_args = True
@ -7447,6 +7509,36 @@ class MapFromEntries(Func):
pass
class MapCat(Func):
arg_types = {"this": True, "expression": True}
class MapContainsKey(Func):
arg_types = {"this": True, "key": True}
class MapDelete(Func):
arg_types = {"this": True, "expressions": True}
is_var_len_args = True
class MapInsert(Func):
arg_types = {"this": True, "key": False, "value": True, "update_flag": False}
class MapKeys(Func):
pass
class MapPick(Func):
arg_types = {"this": True, "expressions": True}
is_var_len_args = True
class MapSize(Func):
pass
# https://learn.microsoft.com/en-us/sql/t-sql/language-elements/scope-resolution-operator-transact-sql?view=sql-server-ver16
class ScopeResolution(Expression):
arg_types = {"this": False, "expression": True}
@ -7492,7 +7584,11 @@ class MD5(Func):
# Represents the variant of the MD5 function that returns a binary value
# Var len args due to Exasol:
# https://docs.exasol.com/db/latest/sql_references/functions/alphabeticallistfunctions/hashtype_md5.htm
class MD5Digest(Func):
arg_types = {"this": True, "expressions": False}
is_var_len_args = True
_sql_names = ["MD5_DIGEST"]
@ -7524,11 +7620,11 @@ class Month(Func):
class Monthname(Func):
pass
arg_types = {"this": True, "abbreviated": False}
class AddMonths(Func):
arg_types = {"this": True, "expression": True}
arg_types = {"this": True, "expression": True, "preserve_end_of_month": False}
class Nvl2(Func):
@ -7976,7 +8072,7 @@ class StrToTime(Func):
# Spark allows unix_timestamp()
# https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.functions.unix_timestamp.html
# https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.unix_timestamp.html
class StrToUnix(Func):
arg_types = {"this": False, "format": False}
@ -8150,6 +8246,7 @@ class UnixToTime(Func):
"hours": False,
"minutes": False,
"format": False,
"target_type": False,
}
SECONDS = Literal.number(0)
@ -8226,7 +8323,10 @@ class Upper(Func):
class Corr(Binary, AggFunc):
pass
# Correlation divides by variance(column). If a column has 0 variance, the denominator
# is 0 - some dialects return NaN (DuckDB) while others return NULL (Snowflake).
# `null_on_zero_variance` is set to True at parse time for dialects that return NULL.
arg_types = {"this": True, "expression": True, "null_on_zero_variance": False}
# https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/CUME_DIST.html
@ -8243,20 +8343,30 @@ class VariancePop(AggFunc):
_sql_names = ["VARIANCE_POP", "VAR_POP"]
class Kurtosis(AggFunc):
pass
class Skewness(AggFunc):
pass
class WidthBucket(Func):
arg_types = {"this": True, "min_value": True, "max_value": True, "num_buckets": True}
arg_types = {
"this": True,
"min_value": False,
"max_value": False,
"num_buckets": False,
"threshold": False,
}
class CovarSamp(Binary, AggFunc):
pass
class CovarSamp(AggFunc):
arg_types = {"this": True, "expression": True}
class CovarPop(Binary, AggFunc):
pass
class CovarPop(AggFunc):
arg_types = {"this": True, "expression": True}
class Week(Func):
@ -8273,7 +8383,7 @@ class NextDay(Func):
class XMLElement(Func):
_sql_names = ["XMLELEMENT"]
arg_types = {"this": True, "expressions": False}
arg_types = {"this": True, "expressions": False, "evalname": False}
class XMLGet(Func):

140
sqlglot/generator.py
View file

@ -112,6 +112,7 @@ class Generator(metaclass=_Generator):
TRANSFORMS: t.Dict[t.Type[exp.Expression], t.Callable[..., str]] = {
**JSON_PATH_PART_TRANSFORMS,
exp.Adjacent: lambda self, e: self.binary(e, "-|-"),
exp.AllowedValuesProperty: lambda self,
e: f"ALLOWED_VALUES {self.expressions(e, flat=True)}",
exp.AnalyzeColumns: lambda self, e: self.sql(e, "this"),
@ -514,6 +515,11 @@ class Generator(metaclass=_Generator):
# Whether to include the VARIABLE keyword for SET assignments
SET_ASSIGNMENT_REQUIRES_VARIABLE_KEYWORD = False
# Whether FROM is supported in UPDATE statements or if joins must be generated instead, e.g:
# Supported (Postgres, Doris etc): UPDATE t1 SET t1.a = t2.b FROM t2
# Unsupported (MySQL, SingleStore): UPDATE t1 JOIN t2 ON TRUE SET t1.a = t2.b
UPDATE_STATEMENT_SUPPORTS_FROM = True
TYPE_MAPPING = {
exp.DataType.Type.DATETIME2: "TIMESTAMP",
exp.DataType.Type.NCHAR: "CHAR",
@ -1152,6 +1158,19 @@ class Generator(metaclass=_Generator):
options = f" {options}" if options else ""
return f"UNIQUE{nulls_sql}{this}{index_type}{on_conflict}{options}"
def inoutcolumnconstraint_sql(self, expression: exp.InOutColumnConstraint) -> str:
input_ = expression.args.get("input_")
output = expression.args.get("output")
if input_ and output:
return "IN OUT"
if input_:
return "IN"
if output:
return "OUT"
return ""
def createable_sql(self, expression: exp.Create, locations: t.DefaultDict) -> str:
return self.sql(expression, "this")
@ -1455,7 +1474,15 @@ class Generator(metaclass=_Generator):
def datatype_sql(self, expression: exp.DataType) -> str:
nested = ""
values = ""
interior = self.expressions(expression, flat=True)
expr_nested = expression.args.get("nested")
interior = (
self.expressions(
expression, dynamic=True, new_line=True, skip_first=True, skip_last=True
)
if expr_nested and self.pretty
else self.expressions(expression, flat=True)
)
type_value = expression.this
if type_value in self.UNSUPPORTED_TYPES:
@ -1473,7 +1500,7 @@ class Generator(metaclass=_Generator):
)
if interior:
if expression.args.get("nested"):
if expr_nested:
nested = f"{self.STRUCT_DELIMITER[0]}{interior}{self.STRUCT_DELIMITER[1]}"
if expression.args.get("values") is not None:
delimiters = ("[", "]") if type_value == exp.DataType.Type.ARRAY else ("(", ")")
@ -2019,7 +2046,12 @@ class Generator(metaclass=_Generator):
constraint = f" ON CONSTRAINT {constraint}" if constraint else ""
conflict_keys = self.expressions(expression, key="conflict_keys", flat=True)
conflict_keys = f"({conflict_keys}) " if conflict_keys else " "
if conflict_keys:
conflict_keys = f"({conflict_keys})"
index_predicate = self.sql(expression, "index_predicate")
conflict_keys = f"{conflict_keys}{index_predicate} "
action = self.sql(expression, "action")
expressions = self.expressions(expression, flat=True)
@ -2222,10 +2254,42 @@ class Generator(metaclass=_Generator):
def tuple_sql(self, expression: exp.Tuple) -> str:
return f"({self.expressions(expression, dynamic=True, new_line=True, skip_first=True, skip_last=True)})"
def _update_from_joins_sql(self, expression: exp.Update) -> t.Tuple[str, str]:
"""
Returns (join_sql, from_sql) for UPDATE statements.
- join_sql: placed after UPDATE table, before SET
- from_sql: placed after SET clause (standard position)
Dialects like MySQL need to convert FROM to JOIN syntax.
"""
if self.UPDATE_STATEMENT_SUPPORTS_FROM or not (from_expr := expression.args.get("from_")):
return ("", self.sql(expression, "from_"))
# Qualify unqualified columns in SET clause with the target table
# MySQL requires qualified column names in multi-table UPDATE to avoid ambiguity
target_table = expression.this
if isinstance(target_table, exp.Table):
target_name = exp.to_identifier(target_table.alias_or_name)
for eq in expression.expressions:
col = eq.this
if isinstance(col, exp.Column) and not col.table:
col.set("table", target_name)
table = from_expr.this
if nested_joins := table.args.get("joins", []):
table.set("joins", None)
join_sql = self.sql(exp.Join(this=table, on=exp.true()))
for nested in nested_joins:
if not nested.args.get("on") and not nested.args.get("using"):
nested.set("on", exp.true())
join_sql += self.sql(nested)
return (join_sql, "")
def update_sql(self, expression: exp.Update) -> str:
this = self.sql(expression, "this")
join_sql, from_sql = self._update_from_joins_sql(expression)
set_sql = self.expressions(expression, flat=True)
from_sql = self.sql(expression, "from_")
where_sql = self.sql(expression, "where")
returning = self.sql(expression, "returning")
order = self.sql(expression, "order")
@ -2236,7 +2300,7 @@ class Generator(metaclass=_Generator):
expression_sql = f"{returning}{from_sql}{where_sql}"
options = self.expressions(expression, key="options")
options = f" OPTION({options})" if options else ""
sql = f"UPDATE {this} SET {set_sql}{expression_sql}{order}{limit}{options}"
sql = f"UPDATE {this}{join_sql} SET {set_sql}{expression_sql}{order}{limit}{options}"
return self.prepend_ctes(expression, sql)
def values_sql(self, expression: exp.Values, values_as_table: bool = True) -> str:
@ -4933,30 +4997,49 @@ class Generator(metaclass=_Generator):
return self.func("JSON_EXISTS", this, path)
def arrayagg_sql(self, expression: exp.ArrayAgg) -> str:
array_agg = self.function_fallback_sql(expression)
def _add_arrayagg_null_filter(
self,
array_agg_sql: str,
array_agg_expr: exp.ArrayAgg,
column_expr: exp.Expression,
) -> str:
"""
Add NULL filter to ARRAY_AGG if dialect requires it.
Args:
array_agg_sql: The generated ARRAY_AGG SQL string
array_agg_expr: The ArrayAgg expression node
column_expr: The column/expression to filter (before ORDER BY wrapping)
Returns:
SQL string with FILTER clause added if needed
"""
# Add a NULL FILTER on the column to mimic the results going from a dialect that excludes nulls
# on ARRAY_AGG (e.g Spark) to one that doesn't (e.g. DuckDB)
if self.dialect.ARRAY_AGG_INCLUDES_NULLS and expression.args.get("nulls_excluded"):
parent = expression.parent
if isinstance(parent, exp.Filter):
parent_cond = parent.expression.this
parent_cond.replace(parent_cond.and_(expression.this.is_(exp.null()).not_()))
else:
this = expression.this
# Do not add the filter if the input is not a column (e.g. literal, struct etc)
if this.find(exp.Column):
# DISTINCT is already present in the agg function, do not propagate it to FILTER as well
this_sql = (
self.expressions(this)
if isinstance(this, exp.Distinct)
else self.sql(expression, "this")
)
if not (
self.dialect.ARRAY_AGG_INCLUDES_NULLS and array_agg_expr.args.get("nulls_excluded")
):
return array_agg_sql
array_agg = f"{array_agg} FILTER(WHERE {this_sql} IS NOT NULL)"
parent = array_agg_expr.parent
if isinstance(parent, exp.Filter):
parent_cond = parent.expression.this
parent_cond.replace(parent_cond.and_(column_expr.is_(exp.null()).not_()))
elif column_expr.find(exp.Column):
# Do not add the filter if the input is not a column (e.g. literal, struct etc)
# DISTINCT is already present in the agg function, do not propagate it to FILTER as well
this_sql = (
self.expressions(column_expr)
if isinstance(column_expr, exp.Distinct)
else self.sql(column_expr)
)
array_agg_sql = f"{array_agg_sql} FILTER(WHERE {this_sql} IS NOT NULL)"
return array_agg
return array_agg_sql
def arrayagg_sql(self, expression: exp.ArrayAgg) -> str:
array_agg = self.function_fallback_sql(expression)
return self._add_arrayagg_null_filter(array_agg, expression, expression.this)
def slice_sql(self, expression: exp.Slice) -> str:
step = self.sql(expression, "step")
@ -5158,7 +5241,8 @@ class Generator(metaclass=_Generator):
return include
def xmlelement_sql(self, expression: exp.XMLElement) -> str:
name = f"NAME {self.sql(expression, 'this')}"
prefix = "EVALNAME" if expression.args.get("evalname") else "NAME"
name = f"{prefix} {self.sql(expression, 'this')}"
return self.func("XMLELEMENT", name, *expression.expressions)
def xmlkeyvalueoption_sql(self, expression: exp.XMLKeyValueOption) -> str:
@ -5471,3 +5555,9 @@ class Generator(metaclass=_Generator):
return "WEEK"
return self.func("WEEK", expression.this)
def chr_sql(self, expression: exp.Chr, name: str = "CHR") -> str:
this = self.expressions(expression)
charset = self.sql(expression, "charset")
using = f" USING {charset}" if charset else ""
return self.func(name, this + using)

64
sqlglot/optimizer/annotate_types.py
View file

@ -536,7 +536,7 @@ class TypeAnnotator(metaclass=_TypeAnnotator):
elif (left_type, right_type) in self.binary_coercions:
self._set_type(expression, self.binary_coercions[(left_type, right_type)](left, right))
else:
self._set_type(expression, self._maybe_coerce(left_type, right_type))
self._annotate_by_args(expression, left, right)
if isinstance(expression, exp.Is) or (
left.meta.get("nonnull") is True and right.meta.get("nonnull") is True
@ -572,28 +572,64 @@ class TypeAnnotator(metaclass=_TypeAnnotator):
def _annotate_by_args(
self,
expression: E,
*args: str,
*args: str | exp.Expression,
promote: bool = False,
array: bool = False,
) -> E:
expressions: t.List[exp.Expression] = []
literal_type = None
non_literal_type = None
nested_type = None
for arg in args:
arg_expr = expression.args.get(arg)
expressions.extend(expr for expr in ensure_list(arg_expr) if expr)
if isinstance(arg, str):
expressions = expression.args.get(arg)
else:
expressions = arg
last_datatype = None
for expr in expressions:
expr_type = expr.type
for expr in ensure_list(expressions):
expr_type = expr.type
# Stop at the first nested data type found - we don't want to _maybe_coerce nested types
if expr_type.args.get("nested"):
last_datatype = expr_type
# Stop at the first nested data type found - we don't want to _maybe_coerce nested types
if expr_type.args.get("nested"):
nested_type = expr_type
break
if isinstance(expr, exp.Literal):
literal_type = self._maybe_coerce(literal_type or expr_type, expr_type)
else:
non_literal_type = self._maybe_coerce(non_literal_type or expr_type, expr_type)
if nested_type:
break
if not expr_type.is_type(exp.DataType.Type.UNKNOWN):
last_datatype = self._maybe_coerce(last_datatype or expr_type, expr_type)
result_type = None
self._set_type(expression, last_datatype)
if nested_type:
result_type = nested_type
elif literal_type and non_literal_type:
if self.dialect.PRIORITIZE_NON_LITERAL_TYPES:
literal_this_type = (
literal_type.this if isinstance(literal_type, exp.DataType) else literal_type
)
non_literal_this_type = (
non_literal_type.this
if isinstance(non_literal_type, exp.DataType)
else non_literal_type
)
if (
literal_this_type in exp.DataType.INTEGER_TYPES
and non_literal_this_type in exp.DataType.INTEGER_TYPES
) or (
literal_this_type in exp.DataType.REAL_TYPES
and non_literal_this_type in exp.DataType.REAL_TYPES
):
result_type = non_literal_type
else:
result_type = literal_type or non_literal_type or exp.DataType.Type.UNKNOWN
self._set_type(
expression, result_type or self._maybe_coerce(non_literal_type, literal_type)
)
if promote:
if expression.type.this in exp.DataType.INTEGER_TYPES:

19
sqlglot/optimizer/resolver.py
View file

@ -305,6 +305,21 @@ class Resolver:
# Performance optimization - avoid copying first_columns if there is only one table.
return SingleValuedMapping(first_columns, first_table)
# For BigQuery UNNEST_COLUMN_ONLY, build a mapping of original UNNEST aliases
# from alias.columns[0] to their source names. This is used to resolve shadowing
# where an UNNEST alias shadows a column name from another table.
unnest_original_aliases: t.Dict[str, str] = {}
if self.dialect.UNNEST_COLUMN_ONLY:
unnest_original_aliases = {
alias_arg.columns[0].name: source_name
for source_name, source in self.scope.sources.items()
if (
isinstance(source.expression, exp.Unnest)
and (alias_arg := source.expression.args.get("alias"))
and alias_arg.columns
)
}
unambiguous_columns = {col: first_table for col in first_columns}
all_columns = set(unambiguous_columns)
@ -314,6 +329,10 @@ class Resolver:
all_columns.update(columns)
for column in ambiguous:
if column in unnest_original_aliases:
unambiguous_columns[column] = unnest_original_aliases[column]
continue
unambiguous_columns.pop(column, None)
for column in unique.difference(ambiguous):
unambiguous_columns[column] = table

165
sqlglot/parser.py
View file

@ -155,12 +155,13 @@ def build_convert_timezone(
return exp.ConvertTimezone.from_arg_list(args)
def build_trim(args: t.List, is_left: bool = True):
return exp.Trim(
this=seq_get(args, 0),
expression=seq_get(args, 1),
position="LEADING" if is_left else "TRAILING",
)
def build_trim(args: t.List, is_left: bool = True, reverse_args: bool = False):
this, expression = seq_get(args, 0), seq_get(args, 1)
if expression and reverse_args:
this, expression = expression, this
return exp.Trim(this=this, expression=expression, position="LEADING" if is_left else "TRAILING")
def build_coalesce(
@ -212,8 +213,6 @@ class Parser(metaclass=_Parser):
"ARRAY_AGG": lambda args, dialect: exp.ArrayAgg(
this=seq_get(args, 0), nulls_excluded=dialect.ARRAY_AGG_INCLUDES_NULLS is None or None
),
"CHAR": lambda args: exp.Chr(expressions=args),
"CHR": lambda args: exp.Chr(expressions=args),
"COUNT": lambda args: exp.Count(this=seq_get(args, 0), expressions=args[1:], big_int=True),
"CONCAT": lambda args, dialect: exp.Concat(
expressions=args,
@ -239,12 +238,23 @@ class Parser(metaclass=_Parser):
is_string=dialect.UUID_IS_STRING_TYPE or None
),
"GLOB": lambda args: exp.Glob(this=seq_get(args, 1), expression=seq_get(args, 0)),
"GREATEST": lambda args: exp.Greatest(this=seq_get(args, 0), expressions=args[1:]),
"LEAST": lambda args: exp.Least(this=seq_get(args, 0), expressions=args[1:]),
"GREATEST": lambda args, dialect: exp.Greatest(
this=seq_get(args, 0),
expressions=args[1:],
ignore_nulls=dialect.LEAST_GREATEST_IGNORES_NULLS,
),
"LEAST": lambda args, dialect: exp.Least(
this=seq_get(args, 0),
expressions=args[1:],
ignore_nulls=dialect.LEAST_GREATEST_IGNORES_NULLS,
),
"HEX": build_hex,
"JSON_EXTRACT": build_extract_json_with_path(exp.JSONExtract),
"JSON_EXTRACT_SCALAR": build_extract_json_with_path(exp.JSONExtractScalar),
"JSON_EXTRACT_PATH_TEXT": build_extract_json_with_path(exp.JSONExtractScalar),
"JSON_KEYS": lambda args, dialect: exp.JSONKeys(
this=seq_get(args, 0), expression=dialect.to_json_path(seq_get(args, 1))
),
"LIKE": build_like,
"LOG": build_logarithm,
"LOG2": lambda args: exp.Log(this=exp.Literal.number(2), expression=seq_get(args, 0)),
@ -979,6 +989,10 @@ class Parser(metaclass=_Parser):
TokenType.QMARK_AMP: binary_range_parser(exp.JSONBContainsAllTopKeys),
TokenType.QMARK_PIPE: binary_range_parser(exp.JSONBContainsAnyTopKeys),
TokenType.HASH_DASH: binary_range_parser(exp.JSONBDeleteAtPath),
TokenType.ADJACENT: binary_range_parser(exp.Adjacent),
TokenType.OPERATOR: lambda self, this: self._parse_operator(this),
TokenType.AMP_LT: binary_range_parser(exp.ExtendsLeft),
TokenType.AMP_GT: binary_range_parser(exp.ExtendsRight),
}
PIPE_SYNTAX_TRANSFORM_PARSERS = {
@ -1269,6 +1283,8 @@ class Parser(metaclass=_Parser):
"CAST": lambda self: self._parse_cast(self.STRICT_CAST),
"CEIL": lambda self: self._parse_ceil_floor(exp.Ceil),
"CONVERT": lambda self: self._parse_convert(self.STRICT_CAST),
"CHAR": lambda self: self._parse_char(),
"CHR": lambda self: self._parse_char(),
"DECODE": lambda self: self._parse_decode(),
"EXTRACT": lambda self: self._parse_extract(),
"FLOOR": lambda self: self._parse_ceil_floor(exp.Floor),
@ -1288,11 +1304,7 @@ class Parser(metaclass=_Parser):
"TRIM": lambda self: self._parse_trim(),
"TRY_CAST": lambda self: self._parse_cast(False, safe=True),
"TRY_CONVERT": lambda self: self._parse_convert(False, safe=True),
"XMLELEMENT": lambda self: self.expression(
exp.XMLElement,
this=self._match_text_seq("NAME") and self._parse_id_var(),
expressions=self._match(TokenType.COMMA) and self._parse_csv(self._parse_expression),
),
"XMLELEMENT": lambda self: self._parse_xml_element(),
"XMLTABLE": lambda self: self._parse_xml_table(),
}
@ -3042,6 +3054,8 @@ class Parser(metaclass=_Parser):
conflict_keys = self._parse_csv(self._parse_id_var)
self._match_r_paren()
index_predicate = self._parse_where()
action = self._parse_var_from_options(self.CONFLICT_ACTIONS)
if self._prev.token_type == TokenType.UPDATE:
self._match(TokenType.SET)
@ -3055,6 +3069,7 @@ class Parser(metaclass=_Parser):
expressions=expressions,
action=action,
conflict_keys=conflict_keys,
index_predicate=index_predicate,
constraint=constraint,
where=self._parse_where(),
)
@ -3172,7 +3187,12 @@ class Parser(metaclass=_Parser):
elif self._match(TokenType.RETURNING, advance=False):
kwargs["returning"] = self._parse_returning()
elif self._match(TokenType.FROM, advance=False):
kwargs["from_"] = self._parse_from(joins=True)
from_ = self._parse_from(joins=True)
table = from_.this if from_ else None
if isinstance(table, exp.Subquery) and self._match(TokenType.JOIN, advance=False):
table.set("joins", list(self._parse_joins()) or None)
kwargs["from_"] = from_
elif self._match(TokenType.WHERE, advance=False):
kwargs["where"] = self._parse_where()
elif self._match(TokenType.ORDER_BY, advance=False):
@ -5184,27 +5204,7 @@ class Parser(metaclass=_Parser):
exp.Escape, this=this, expression=self._parse_string() or self._parse_null()
)
def _parse_interval(self, match_interval: bool = True) -> t.Optional[exp.Add | exp.Interval]:
index = self._index
if not self._match(TokenType.INTERVAL) and match_interval:
return None
if self._match(TokenType.STRING, advance=False):
this = self._parse_primary()
else:
this = self._parse_term()
if not this or (
isinstance(this, exp.Column)
and not this.table
and not this.this.quoted
and self._curr
and self._curr.text.upper() not in self.dialect.VALID_INTERVAL_UNITS
):
self._retreat(index)
return None
def _parse_interval_span(self, this: exp.Expression) -> exp.Interval:
# handle day-time format interval span with omitted units:
# INTERVAL '<number days> hh[:][mm[:ss[.ff]]]' <maybe `unit TO unit`>
interval_span_units_omitted = None
@ -5255,10 +5255,35 @@ class Parser(metaclass=_Parser):
if self.INTERVAL_SPANS and self._match_text_seq("TO"):
unit = self.expression(
exp.IntervalSpan, this=unit, expression=self._parse_var(any_token=True, upper=True)
exp.IntervalSpan,
this=unit,
expression=self._parse_function() or self._parse_var(any_token=True, upper=True),
)
interval = self.expression(exp.Interval, this=this, unit=unit)
return self.expression(exp.Interval, this=this, unit=unit)
def _parse_interval(self, match_interval: bool = True) -> t.Optional[exp.Add | exp.Interval]:
index = self._index
if not self._match(TokenType.INTERVAL) and match_interval:
return None
if self._match(TokenType.STRING, advance=False):
this = self._parse_primary()
else:
this = self._parse_term()
if not this or (
isinstance(this, exp.Column)
and not this.table
and not this.this.quoted
and self._curr
and self._curr.text.upper() not in self.dialect.VALID_INTERVAL_UNITS
):
self._retreat(index)
return None
interval = self._parse_interval_span(this)
index = self._index
self._match(TokenType.PLUS)
@ -5590,8 +5615,8 @@ class Parser(metaclass=_Parser):
elif self._match_text_seq("WITHOUT", "TIME", "ZONE"):
maybe_func = False
elif type_token == TokenType.INTERVAL:
unit = self._parse_var(upper=True)
if unit:
if self._curr and self._curr.text.upper() in self.dialect.VALID_INTERVAL_UNITS:
unit = self._parse_var(upper=True)
if self._match_text_seq("TO"):
unit = exp.IntervalSpan(this=unit, expression=self._parse_var(upper=True))
@ -6232,6 +6257,14 @@ class Parser(metaclass=_Parser):
not_null=self._match_pair(TokenType.NOT, TokenType.NULL),
)
constraints.append(self.expression(exp.ColumnConstraint, kind=constraint_kind))
elif not kind and self._match_set({TokenType.IN, TokenType.OUT}, advance=False):
in_out_constraint = self.expression(
exp.InOutColumnConstraint,
input_=self._match(TokenType.IN),
output=self._match(TokenType.OUT),
)
constraints.append(in_out_constraint)
kind = self._parse_types()
elif (
kind
and self._match(TokenType.ALIAS, advance=False)
@ -6517,7 +6550,7 @@ class Parser(metaclass=_Parser):
)
def _parse_primary_key_part(self) -> t.Optional[exp.Expression]:
return self._parse_ordered() or self._parse_field()
return self._parse_field()
def _parse_period_for_system_time(self) -> t.Optional[exp.PeriodForSystemTimeConstraint]:
if not self._match(TokenType.TIMESTAMP_SNAPSHOT):
@ -6748,6 +6781,13 @@ class Parser(metaclass=_Parser):
gap_fill = exp.GapFill.from_arg_list(args)
return self.validate_expression(gap_fill, args)
def _parse_char(self) -> exp.Chr:
return self.expression(
exp.Chr,
expressions=self._parse_csv(self._parse_assignment),
charset=self._match(TokenType.USING) and self._parse_var(),
)
def _parse_cast(self, strict: bool, safe: t.Optional[bool] = None) -> exp.Expression:
this = self._parse_disjunction()
@ -6865,7 +6905,7 @@ class Parser(metaclass=_Parser):
if self._match(TokenType.USING):
to: t.Optional[exp.Expression] = self.expression(
exp.CharacterSet, this=self._parse_var()
exp.CharacterSet, this=self._parse_var(tokens={TokenType.BINARY})
)
elif self._match(TokenType.COMMA):
to = self._parse_types()
@ -6874,6 +6914,22 @@ class Parser(metaclass=_Parser):
return self.build_cast(strict=strict, this=this, to=to, safe=safe)
def _parse_xml_element(self) -> exp.XMLElement:
if self._match_text_seq("EVALNAME"):
evalname = True
this = self._parse_bitwise()
else:
evalname = None
self._match_text_seq("NAME")
this = self._parse_id_var()
return self.expression(
exp.XMLElement,
this=this,
expressions=self._match(TokenType.COMMA) and self._parse_csv(self._parse_bitwise),
evalname=evalname,
)
def _parse_xml_table(self) -> exp.XMLTable:
namespaces = None
passing = None
@ -9078,3 +9134,26 @@ class Parser(metaclass=_Parser):
expr.set("expression", exp.Literal.string(self.dialect.INITCAP_DEFAULT_DELIMITER_CHARS))
return expr
def _parse_operator(self, this: t.Optional[exp.Expression]) -> t.Optional[exp.Expression]:
while True:
if not self._match(TokenType.L_PAREN):
break
op = ""
while self._curr and not self._match(TokenType.R_PAREN):
op += self._curr.text
self._advance()
this = self.expression(
exp.Operator,
comments=self._prev_comments,
this=this,
operator=op,
expression=self._parse_bitwise(),
)
if not self._match(TokenType.OPERATOR):
break
return this

1
sqlglot/schema.py
View file

@ -471,6 +471,7 @@ class MappingSchema(AbstractMappingSchema, Schema):
try:
expression = exp.DataType.build(schema_type, dialect=dialect, udt=udt)
expression.transform(dialect.normalize_identifier, copy=False)
self._type_mapping_cache[schema_type] = expression
except AttributeError:
in_dialect = f" in dialect {dialect}" if dialect else ""

16
sqlglot/tokens.py
View file

@ -87,6 +87,7 @@ class TokenType(AutoName):
DAMP = auto()
AMP_LT = auto()
AMP_GT = auto()
ADJACENT = auto()
XOR = auto()
DSTAR = auto()
QMARK_AMP = auto()
@ -207,6 +208,7 @@ class TokenType(AutoName):
LINESTRING = auto()
LOCALTIME = auto()
LOCALTIMESTAMP = auto()
SYSTIMESTAMP = auto()
MULTILINESTRING = auto()
POLYGON = auto()
MULTIPOLYGON = auto()
@ -370,6 +372,7 @@ class TokenType(AutoName):
ORDER_SIBLINGS_BY = auto()
ORDERED = auto()
ORDINALITY = auto()
OUT = auto()
OUTER = auto()
OVER = auto()
OVERLAPS = auto()
@ -714,6 +717,8 @@ class Tokenizer(metaclass=_Tokenizer):
**{f"{{{{{postfix}": TokenType.BLOCK_START for postfix in ("+", "-")},
**{f"{prefix}}}}}": TokenType.BLOCK_END for prefix in ("+", "-")},
HINT_START: TokenType.HINT,
"&<": TokenType.AMP_LT,
"&>": TokenType.AMP_GT,
"==": TokenType.EQ,
"::": TokenType.DCOLON,
"?::": TokenType.QDCOLON,
@ -737,6 +742,7 @@ class Tokenizer(metaclass=_Tokenizer):
"~~": TokenType.LIKE,
"~~*": TokenType.ILIKE,
"~*": TokenType.IRLIKE,
"-|-": TokenType.ADJACENT,
"ALL": TokenType.ALL,
"AND": TokenType.AND,
"ANTI": TokenType.ANTI,
@ -837,6 +843,7 @@ class Tokenizer(metaclass=_Tokenizer):
"XOR": TokenType.XOR,
"ORDER BY": TokenType.ORDER_BY,
"ORDINALITY": TokenType.ORDINALITY,
"OUT": TokenType.OUT,
"OUTER": TokenType.OUTER,
"OVER": TokenType.OVER,
"OVERLAPS": TokenType.OVERLAPS,
@ -850,6 +857,7 @@ class Tokenizer(metaclass=_Tokenizer):
"PRAGMA": TokenType.PRAGMA,
"PRIMARY KEY": TokenType.PRIMARY_KEY,
"PROCEDURE": TokenType.PROCEDURE,
"OPERATOR": TokenType.OPERATOR,
"QUALIFY": TokenType.QUALIFY,
"RANGE": TokenType.RANGE,
"RECURSIVE": TokenType.RECURSIVE,
@ -1363,8 +1371,12 @@ class Tokenizer(metaclass=_Tokenizer):
decimal = True
self._advance()
elif self._peek in ("-", "+") and scientific == 1:
scientific += 1
self._advance()
# Only consume +/- if followed by a digit
if self._current + 1 < self.size and self.sql[self._current + 1].isdigit():
scientific += 1
self._advance()
else:
return self._add(TokenType.NUMBER)
elif self._peek.upper() == "E" and not scientific:
scientific += 1
self._advance()

17
sqlglot/typing/__init__.py
View file

@ -47,10 +47,13 @@ EXPRESSION_METADATA: ExpressionMetadataType = {
**{
expr_type: {"returns": exp.DataType.Type.BOOLEAN}
for expr_type in {
exp.All,
exp.Any,
exp.Between,
exp.Boolean,
exp.Contains,
exp.EndsWith,
exp.Exists,
exp.In,
exp.LogicalAnd,
exp.LogicalOr,
@ -141,6 +144,7 @@ EXPRESSION_METADATA: ExpressionMetadataType = {
expr_type: {"returns": exp.DataType.Type.TIME}
for expr_type in {
exp.CurrentTime,
exp.Localtime,
exp.Time,
exp.TimeAdd,
exp.TimeSub,
@ -200,6 +204,7 @@ EXPRESSION_METADATA: ExpressionMetadataType = {
exp.UnixToStr,
exp.UnixToTimeStr,
exp.Upper,
exp.RawString,
}
},
**{
@ -269,7 +274,11 @@ EXPRESSION_METADATA: ExpressionMetadataType = {
exp.Array: {"annotator": lambda self, e: self._annotate_by_args(e, "expressions", array=True)},
exp.ArrayAgg: {"annotator": lambda self, e: self._annotate_by_args(e, "this", array=True)},
exp.Bracket: {"annotator": lambda self, e: self._annotate_bracket(e)},
exp.Case: {"annotator": lambda self, e: self._annotate_by_args(e, "default", "ifs")},
exp.Case: {
"annotator": lambda self, e: self._annotate_by_args(
e, *[if_expr.args["true"] for if_expr in e.args["ifs"]], "default"
)
},
exp.Count: {
"annotator": lambda self, e: self._set_type(
e, exp.DataType.Type.BIGINT if e.args.get("big_int") else exp.DataType.Type.INT
@ -286,6 +295,12 @@ EXPRESSION_METADATA: ExpressionMetadataType = {
exp.Dot: {"annotator": lambda self, e: self._annotate_dot(e)},
exp.Explode: {"annotator": lambda self, e: self._annotate_explode(e)},
exp.Extract: {"annotator": lambda self, e: self._annotate_extract(e)},
exp.HexString: {
"annotator": lambda self, e: self._set_type(
e,
exp.DataType.Type.BIGINT if e.args.get("is_integer") else exp.DataType.Type.BINARY,
)
},
exp.GenerateSeries: {
"annotator": lambda self, e: self._annotate_by_args(e, "start", "end", "step", array=True)
},

3
sqlglot/typing/hive.py
View file

@ -9,4 +9,7 @@ EXPRESSION_METADATA = {
exp.Coalesce: {
"annotator": lambda self, e: self._annotate_by_args(e, "this", "expressions", promote=True)
},
exp.Encode: {"returns": exp.DataType.Type.BINARY},
exp.StrToUnix: {"returns": exp.DataType.Type.BIGINT},
exp.Tanh: {"returns": exp.DataType.Type.DOUBLE},
}

10
sqlglot/typing/mysql.py Normal file
View file

@ -0,0 +1,10 @@
from __future__ import annotations
from sqlglot import exp
from sqlglot.typing import EXPRESSION_METADATA
EXPRESSION_METADATA = {
**EXPRESSION_METADATA,
**{expr_type: {"returns": exp.DataType.Type.VARCHAR} for expr_type in (exp.Elt,)},
exp.Localtime: {"returns": exp.DataType.Type.DATETIME},
}

59
sqlglot/typing/snowflake.py
View file

@ -180,6 +180,29 @@ def _annotate_variance(self: TypeAnnotator, expression: exp.Expression) -> exp.E
return expression
def _annotate_kurtosis(self: TypeAnnotator, expression: exp.Kurtosis) -> exp.Kurtosis:
"""Annotate KURTOSIS with correct return type.
Based on Snowflake behavior:
- DECFLOAT input -> DECFLOAT
- DOUBLE or FLOAT input -> DOUBLE
- Other numeric types (INT, NUMBER) -> NUMBER(38, 12)
"""
expression = self._annotate_by_args(expression, "this")
input_type = expression.this.type
if input_type.is_type(exp.DataType.Type.DECFLOAT):
self._set_type(expression, exp.DataType.build("DECFLOAT", dialect="snowflake"))
elif input_type.is_type(exp.DataType.Type.FLOAT, exp.DataType.Type.DOUBLE):
self._set_type(expression, exp.DataType.Type.DOUBLE)
else:
self._set_type(
expression, exp.DataType.build(f"NUMBER({MAX_PRECISION}, 12)", dialect="snowflake")
)
return expression
def _annotate_math_with_float_decfloat(
self: TypeAnnotator, expression: exp.Expression
) -> exp.Expression:
@ -239,10 +262,16 @@ EXPRESSION_METADATA = {
for expr_type in (
exp.ApproxTopK,
exp.ApproxTopKEstimate,
exp.Array,
exp.ArrayAgg,
exp.ArrayAppend,
exp.ArrayConcat,
exp.ArrayConstructCompact,
exp.ArrayPrepend,
exp.ArrayRemove,
exp.ArrayUniqueAgg,
exp.ArrayUnionAgg,
exp.MapKeys,
exp.RegexpExtractAll,
exp.Split,
exp.StringToArray,
@ -270,6 +299,10 @@ EXPRESSION_METADATA = {
exp.BitmapOrAgg,
exp.Compress,
exp.DecompressBinary,
exp.Decrypt,
exp.DecryptRaw,
exp.Encrypt,
exp.EncryptRaw,
exp.HexString,
exp.MD5Digest,
exp.SHA1Digest,
@ -289,6 +322,7 @@ EXPRESSION_METADATA = {
exp.BoolxorAgg,
exp.EqualNull,
exp.IsNullValue,
exp.MapContainsKey,
exp.Search,
exp.SearchIp,
exp.ToBoolean,
@ -326,6 +360,8 @@ EXPRESSION_METADATA = {
exp.Cbrt,
exp.Cosh,
exp.CosineDistance,
exp.CovarPop,
exp.CovarSamp,
exp.DotProduct,
exp.EuclideanDistance,
exp.ManhattanDistance,
@ -334,6 +370,7 @@ EXPRESSION_METADATA = {
exp.Sinh,
}
},
exp.Kurtosis: {"annotator": _annotate_kurtosis},
**{
expr_type: {"returns": exp.DataType.Type.DECFLOAT}
for expr_type in {
@ -385,6 +422,7 @@ EXPRESSION_METADATA = {
exp.JarowinklerSimilarity,
exp.Length,
exp.Levenshtein,
exp.MapSize,
exp.Minute,
exp.RtrimmedLength,
exp.Second,
@ -406,6 +444,15 @@ EXPRESSION_METADATA = {
exp.XMLGet,
}
},
**{
expr_type: {"returns": exp.DataType.Type.MAP}
for expr_type in {
exp.MapCat,
exp.MapDelete,
exp.MapInsert,
exp.MapPick,
}
},
**{
expr_type: {"returns": exp.DataType.Type.FILE}
for expr_type in {
@ -419,6 +466,13 @@ EXPRESSION_METADATA = {
exp.TsOrDsToTime,
}
},
**{
expr_type: {"returns": exp.DataType.Type.TIMESTAMPLTZ}
for expr_type in {
exp.CurrentTimestamp,
exp.Localtimestamp,
}
},
**{
expr_type: {"returns": exp.DataType.Type.VARCHAR}
for expr_type in {
@ -453,7 +507,6 @@ EXPRESSION_METADATA = {
exp.DecompressString,
exp.HexDecodeString,
exp.HexEncode,
exp.Initcap,
exp.MD5,
exp.Monthname,
exp.Randstr,
@ -500,15 +553,11 @@ EXPRESSION_METADATA = {
},
exp.DateAdd: {"annotator": _annotate_date_or_time_add},
exp.DecodeCase: {"annotator": _annotate_decode_case},
exp.GreatestIgnoreNulls: {
"annotator": lambda self, e: self._annotate_by_args(e, "expressions")
},
exp.HashAgg: {
"annotator": lambda self, e: self._set_type(
e, exp.DataType.build("NUMBER(19, 0)", dialect="snowflake")
)
},
exp.LeastIgnoreNulls: {"annotator": lambda self, e: self._annotate_by_args(e, "expressions")},
exp.Median: {"annotator": _annotate_median},
exp.Reverse: {"annotator": _annotate_reverse},
exp.StrToTime: {"annotator": _annotate_str_to_time},

10
sqlglot/typing/spark.py Normal file
View file

@ -0,0 +1,10 @@
from __future__ import annotations
from sqlglot import exp
from sqlglot.typing.spark2 import EXPRESSION_METADATA
EXPRESSION_METADATA = {
**EXPRESSION_METADATA,
exp.CurrentTimezone: {"returns": exp.DataType.Type.VARCHAR},
exp.Localtimestamp: {"returns": exp.DataType.Type.TIMESTAMPNTZ},
}

6
sqlglot/typing/spark2.py
View file

@ -55,4 +55,10 @@ EXPRESSION_METADATA: ExpressionMetadataType = {
self, e, "this", "fill_pattern", target_type=exp.DataType.Type.TEXT
)
},
**{
expr_type: {"returns": exp.DataType.Type.DOUBLE}
for expr_type in {
exp.Atan2,
}
},
}

2
sqlglotrs/Cargo.lock generated
View file

@ -534,7 +534,7 @@ checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
[[package]]
name = "sqlglotrs"
version = "0.10.0"
version = "0.11.0"
dependencies = [
"criterion",
"pyo3",

2
sqlglotrs/Cargo.toml
View file

@ -1,6 +1,6 @@
[package]
name = "sqlglotrs"
version = "0.10.0"
version = "0.11.0"
edition = "2021"
license = "MIT"

9
sqlglotrs/src/tokenizer.rs
View file

@ -513,8 +513,13 @@ impl<'a> TokenizerState<'a> {
decimal = true;
self.advance(1)?;
} else if (self.peek_char == '-' || self.peek_char == '+') && scientific == 1 {
scientific += 1;
self.advance(1)?;
// Only consume +/- if followed by a digit
if self.current + 1 < self.size && self.sql[self.current + 1].is_ascii_digit() {
scientific += 1;
self.advance(1)?;
} else {
return self.add(self.token_types.number, None);
}
} else if self.peek_char.to_ascii_uppercase() == 'E' && scientific == 0 {
scientific += 1;
self.advance(1)?;

2
tests/dialects/test_bigquery.py
View file

@ -175,6 +175,7 @@ class TestBigQuery(Validator):
self.validate_identity("BEGIN DECLARE y INT64", check_command_warning=True)
self.validate_identity("LOOP SET x = x + 1", check_command_warning=True)
self.validate_identity("REPEAT SET x = x + 1", check_command_warning=True)
self.validate_identity("SELECT MAKE_INTERVAL(100, 11, 1, 12, 30, 10)")
self.validate_identity(
"WHILE i < ARRAY_LENGTH(batches) DO SET x = batches[OFFSET(i)]",
check_command_warning=True,
@ -655,7 +656,6 @@ LANGUAGE js AS
"snowflake": "SELECT LAST_DAY(CAST('2008-11-25' AS DATE), QUARTER)",
},
write={
"duckdb": UnsupportedError,
"bigquery": "SELECT LAST_DAY(CAST('2008-11-25' AS DATE), QUARTER)",
"snowflake": "SELECT LAST_DAY(CAST('2008-11-25' AS DATE), QUARTER)",
},

5
tests/dialects/test_clickhouse.py
View file

@ -443,6 +443,7 @@ class TestClickhouse(Validator):
"mysql": "SELECT 1 XOR 0 XOR 1",
},
)
self.validate_identity("SELECT xor(0, 1, 1, 0)")
self.validate_all(
"CONCAT(a, b)",
read={
@ -675,6 +676,10 @@ class TestClickhouse(Validator):
"SELECT quantilesExactExclusive(0.25, 0.5, 0.75)(x) AS y FROM (SELECT number AS x FROM num)"
)
self.validate_identity("SELECT or(0, 1, -2)", "SELECT 0 OR 1 OR -2")
self.validate_identity("SELECT and(1, 2, 3)", "SELECT 1 AND 2 AND 3")
self.validate_identity("SELECT or(and(3, 0), 5)", "SELECT (3 AND 0) OR 5")
def test_clickhouse_values(self):
ast = self.parse_one("SELECT * FROM VALUES (1, 2, 3)")
self.assertEqual(len(list(ast.find_all(exp.Tuple))), 4)

1
tests/dialects/test_databricks.py
View file

@ -241,6 +241,7 @@ class TestDatabricks(Validator):
self.validate_identity("CURRENT_TIMEZONE()")
self.validate_identity("CURDATE()", "CURRENT_DATE")
self.validate_identity("CURDATE", "CURRENT_DATE")
self.validate_identity("SELECT MAKE_INTERVAL(100, 11, 12, 13, 14, 14, 15)")
# https://docs.databricks.com/sql/language-manual/functions/colonsign.html
def test_json(self):

177
tests/dialects/test_dialect.py
View file

@ -1473,6 +1473,44 @@ class TestDialect(Validator):
},
)
self.validate_all(
"ARRAY_PREPEND(arr, x)",
read={
"": "ARRAY_PREPEND(arr, x)",
"duckdb": "LIST_PREPEND(x, arr)",
"postgres": "ARRAY_PREPEND(x, arr)",
"spark": "ARRAY_PREPEND(arr, x)",
"databricks": "ARRAY_PREPEND(arr, x)",
"snowflake": "ARRAY_PREPEND(arr, x)",
},
write={
"duckdb": "LIST_PREPEND(x, arr)",
"postgres": "ARRAY_PREPEND(x, arr)",
"spark": "ARRAY_PREPEND(arr, x)",
"databricks": "ARRAY_PREPEND(arr, x)",
"snowflake": "ARRAY_PREPEND(arr, x)",
},
)
self.validate_all(
"ARRAY_APPEND(arr, x)",
read={
"": "ARRAY_APPEND(arr, x)",
"duckdb": "LIST_APPEND(arr, x)",
"postgres": "ARRAY_APPEND(arr, x)",
"spark": "ARRAY_APPEND(arr, x)",
"databricks": "ARRAY_APPEND(arr, x)",
"snowflake": "ARRAY_APPEND(arr, x)",
},
write={
"duckdb": "LIST_APPEND(arr, x)",
"postgres": "ARRAY_APPEND(arr, x)",
"spark": "ARRAY_APPEND(arr, x)",
"databricks": "ARRAY_APPEND(arr, x)",
"snowflake": "ARRAY_APPEND(arr, x)",
},
)
def test_order_by(self):
self.validate_identity(
"SELECT c FROM t ORDER BY a, b,",
@ -3383,7 +3421,7 @@ FROM subquery2""",
"spark": "SELECT 1 EXCEPT ALL SELECT 1",
"sqlite": UnsupportedError,
"starrocks": UnsupportedError,
"trino": UnsupportedError,
"trino": "SELECT 1 EXCEPT ALL SELECT 1",
"tsql": UnsupportedError,
},
)
@ -3436,44 +3474,118 @@ FROM subquery2""",
read={
"bigquery": "TRIM('abc', 'a')",
"snowflake": "TRIM('abc', 'a')",
"hive": "TRIM('abc', 'a')",
"spark2": "TRIM('a', 'abc')",
"spark": "TRIM('a', 'abc')",
"databricks": "TRIM('a', 'abc')",
},
write={
"bigquery": "TRIM('abc', 'a')",
"snowflake": "TRIM('abc', 'a')",
"hive": "TRIM('a' FROM 'abc')",
"spark2": "TRIM('a' FROM 'abc')",
"spark": "TRIM('a' FROM 'abc')",
"databricks": "TRIM('a' FROM 'abc')",
},
)
self.validate_all(
"LTRIM('Hello World', 'H')",
read={
"": "LTRIM('Hello World', 'H')",
"oracle": "LTRIM('Hello World', 'H')",
"clickhouse": "TRIM(LEADING 'H' FROM 'Hello World')",
"snowflake": "LTRIM('Hello World', 'H')",
"bigquery": "LTRIM('Hello World', 'H')",
"": "LTRIM('Hello World', 'H')",
"hive": "LTRIM('Hello World', 'H')",
"spark2": "LTRIM('H', 'Hello World')",
"spark": "LTRIM('H', 'Hello World')",
"databricks": "LTRIM('H', 'Hello World')",
},
write={
"clickhouse": "TRIM(LEADING 'H' FROM 'Hello World')",
"oracle": "LTRIM('Hello World', 'H')",
"snowflake": "LTRIM('Hello World', 'H')",
"bigquery": "LTRIM('Hello World', 'H')",
"hive": "TRIM(LEADING 'H' FROM 'Hello World')",
"spark2": "TRIM(LEADING 'H' FROM 'Hello World')",
"spark": "TRIM(LEADING 'H' FROM 'Hello World')",
"databricks": "TRIM(LEADING 'H' FROM 'Hello World')",
},
)
self.validate_all(
"RTRIM('Hello World', 'd')",
read={
"": "RTRIM('Hello World', 'd')",
"clickhouse": "TRIM(TRAILING 'd' FROM 'Hello World')",
"oracle": "RTRIM('Hello World', 'd')",
"snowflake": "RTRIM('Hello World', 'd')",
"bigquery": "RTRIM('Hello World', 'd')",
"": "RTRIM('Hello World', 'd')",
"hive": "RTRIM('Hello World', 'd')",
"spark2": "RTRIM('d', 'Hello World')",
"spark": "RTRIM('d', 'Hello World')",
"databricks": "RTRIM('d', 'Hello World')",
},
write={
"clickhouse": "TRIM(TRAILING 'd' FROM 'Hello World')",
"oracle": "RTRIM('Hello World', 'd')",
"snowflake": "RTRIM('Hello World', 'd')",
"bigquery": "RTRIM('Hello World', 'd')",
"hive": "TRIM(TRAILING 'd' FROM 'Hello World')",
"spark2": "TRIM(TRAILING 'd' FROM 'Hello World')",
"spark": "TRIM(TRAILING 'd' FROM 'Hello World')",
"databricks": "TRIM(TRAILING 'd' FROM 'Hello World')",
},
)
self.validate_all(
"LTRIM('Hello World')",
read={
"": "LTRIM('Hello World')",
"clickhouse": "LTRIM('Hello World')",
"oracle": "LTRIM('Hello World')",
"snowflake": "LTRIM('Hello World')",
"bigquery": "LTRIM('Hello World')",
"hive": "LTRIM('Hello World')",
"spark2": "LTRIM('Hello World')",
"spark": "LTRIM('Hello World')",
"databricks": "LTRIM('Hello World')",
},
write={
"clickhouse": "LTRIM('Hello World')",
"oracle": "LTRIM('Hello World')",
"snowflake": "LTRIM('Hello World')",
"bigquery": "LTRIM('Hello World')",
"hive": "LTRIM('Hello World')",
"spark2": "LTRIM('Hello World')",
"spark": "LTRIM('Hello World')",
"databricks": "LTRIM('Hello World')",
},
)
self.validate_all(
"RTRIM('Hello World')",
read={
"": "RTRIM('Hello World')",
"clickhouse": "RTRIM('Hello World')",
"oracle": "RTRIM('Hello World')",
"snowflake": "RTRIM('Hello World')",
"bigquery": "RTRIM('Hello World')",
"hive": "RTRIM('Hello World')",
"spark2": "RTRIM('Hello World')",
"spark": "RTRIM('Hello World')",
"databricks": "RTRIM('Hello World')",
},
write={
"clickhouse": "RTRIM('Hello World')",
"oracle": "RTRIM('Hello World')",
"snowflake": "RTRIM('Hello World')",
"bigquery": "RTRIM('Hello World')",
"hive": "RTRIM('Hello World')",
"spark2": "RTRIM('Hello World')",
"spark": "RTRIM('Hello World')",
"databricks": "RTRIM('Hello World')",
},
)
@ -4703,7 +4815,6 @@ FROM subquery2""",
"postgres": f"SELECT {func}",
"duckdb": f"SELECT {func}",
"redshift": f"SELECT {func}",
"snowflake": f"SELECT {func}",
"presto": f"SELECT {func}",
"trino": f"SELECT {func}",
"mysql": f"SELECT {func}",
@ -4724,7 +4835,6 @@ FROM subquery2""",
"postgres": f"SELECT {func}(2)",
"duckdb": f"SELECT {func}(2)",
"redshift": f"SELECT {func}(2)",
"snowflake": f"SELECT {func}(2)",
"presto": f"SELECT {func}(2)",
"trino": f"SELECT {func}(2)",
"mysql": f"SELECT {func}(2)",
@ -4840,3 +4950,60 @@ FROM subquery2""",
self.assertEqual(
parse_one(func_sql, dialect=dialect).sql(dialect), no_paren_sql
)
def test_operator(self):
expr = self.validate_identity("1 OPERATOR(+) 2 OPERATOR(*) 3")
expr.left.assert_is(exp.Operator)
expr.left.left.assert_is(exp.Literal)
expr.left.right.assert_is(exp.Literal)
expr.right.assert_is(exp.Literal)
self.assertEqual(expr.sql(dialect="postgres"), "1 OPERATOR(+) 2 OPERATOR(*) 3")
self.validate_identity("SELECT operator FROM t")
self.validate_identity("SELECT 1 OPERATOR(+) 2")
self.validate_identity("SELECT 1 OPERATOR(+) /* foo */ 2")
self.validate_identity("SELECT 1 OPERATOR(pg_catalog.+) 2")
def test_json_keys(self):
self.validate_all(
"JSON_KEYS(foo)",
read={
"": "JSON_KEYS(foo)",
"spark": "JSON_OBJECT_KEYS(foo)",
"databricks": "JSON_OBJECT_KEYS(foo)",
"mysql": "JSON_KEYS(foo)",
"starrocks": "JSON_KEYS(foo)",
"duckdb": "JSON_KEYS(foo)",
"snowflake": "OBJECT_KEYS(foo)",
"doris": "JSON_KEYS(foo)",
"singlestore": "JSON_KEYS(foo)",
},
write={
"spark": "JSON_OBJECT_KEYS(foo)",
"databricks": "JSON_OBJECT_KEYS(foo)",
"mysql": "JSON_KEYS(foo)",
"starrocks": "JSON_KEYS(foo)",
"duckdb": "JSON_KEYS(foo)",
"snowflake": "OBJECT_KEYS(foo)",
"doris": "JSON_KEYS(foo)",
"singlestore": "JSON_KEYS(foo)",
},
)
self.validate_all(
"JSON_KEYS(foo, '$.a')",
read={
"": "JSON_KEYS(foo, '$.a')",
"mysql": "JSON_KEYS(foo, '$.a')",
"starrocks": "JSON_KEYS(foo, '$.a')",
"duckdb": "JSON_KEYS(foo, '$.a')",
"doris": "JSON_KEYS(foo, '$.a')",
},
write={
"mysql": "JSON_KEYS(foo, '$.a')",
"starrocks": "JSON_KEYS(foo, '$.a')",
"duckdb": "JSON_KEYS(foo, '$.a')",
"doris": "JSON_KEYS(foo, '$.a')",
},
)

1
tests/dialects/test_druid.py
View file

@ -14,6 +14,7 @@ class TestDruid(Validator):
self.validate_identity("SELECT FLOOR(col) FROM t")
self.validate_identity("SELECT FLOOR(price, 2) AS rounded_price FROM t")
self.validate_identity("SELECT CURRENT_TIMESTAMP")
self.validate_identity("SELECT ARRAY[1, 2, 3]")
# validate across all dialects
write = {dialect.value: "FLOOR(__time TO WEEK)" for dialect in Dialects}

135
tests/dialects/test_duckdb.py
View file

@ -155,6 +155,7 @@ class TestDuckDB(Validator):
"bigquery": "CREATE TEMP FUNCTION f1(a INT64, b INT64) AS (a + b)",
},
)
self.validate_identity("SELECT GET_BIT(CAST('0110010' AS BIT), 2)")
self.validate_identity("SELECT 1 WHERE x > $1")
self.validate_identity("SELECT 1 WHERE x > $name")
self.validate_identity("""SELECT '{"x": 1}' -> c FROM t""")
@ -369,6 +370,12 @@ class TestDuckDB(Validator):
parse_one("a // b", read="duckdb").assert_is(exp.IntDiv).sql(dialect="duckdb"), "a // b"
)
self.validate_identity(
"SELECT tbl.x*1e4+tbl.y FROM tbl",
"SELECT tbl.x * 1e4 + tbl.y FROM tbl",
)
self.validate_identity("DAYNAME(x)")
self.validate_identity("MONTHNAME(x)")
self.validate_identity(
"SELECT LIST_TRANSFORM([5, NULL, 6], (x, y) -> COALESCE(x, y, 0) + 1)"
)
@ -492,7 +499,7 @@ class TestDuckDB(Validator):
)
self.validate_identity(
"SELECT a, LOGICAL_OR(b) FROM foo GROUP BY a",
"SELECT a, BOOL_OR(b) FROM foo GROUP BY a",
"SELECT a, BOOL_OR(CAST(b AS BOOLEAN)) FROM foo GROUP BY a",
)
self.validate_identity(
"SELECT JSON_EXTRACT_STRING(c, '$.k1') = 'v1'",
@ -542,6 +549,24 @@ class TestDuckDB(Validator):
"JSON_EXTRACT_PATH_TEXT(x, '$.family')",
"x ->> '$.family'",
)
self.validate_all(
"SELECT NOT (data -> '$.value')",
read={
"snowflake": "SELECT NOT data:value",
},
)
self.validate_all(
"SELECT NOT (data -> '$.value.nested')",
read={
"snowflake": "SELECT NOT data:value:nested",
},
)
self.validate_all(
"SELECT (data -> '$.value') = 1",
read={
"snowflake": "SELECT data:value = 1",
},
)
self.validate_identity(
"SELECT {'yes': 'duck', 'maybe': 'goose', 'huh': NULL, 'no': 'heron'}"
)
@ -718,23 +743,6 @@ class TestDuckDB(Validator):
"duckdb": "CREATE TABLE IF NOT EXISTS t (cola INT, colb TEXT)",
},
)
expr = self.parse_one("TO_BINARY('48454C50', 'HEX')", dialect="snowflake")
annotated = annotate_types(expr, dialect="snowflake")
self.assertEqual(annotated.sql("duckdb"), "UNHEX('48454C50')")
expr = self.parse_one("TO_BINARY('48454C50')", dialect="snowflake")
annotated = annotate_types(expr, dialect="snowflake")
self.assertEqual(annotated.sql("duckdb"), "UNHEX('48454C50')")
expr = self.parse_one("TO_BINARY('TEST', 'UTF-8')", dialect="snowflake")
annotated = annotate_types(expr, dialect="snowflake")
self.assertEqual(annotated.sql("duckdb"), "ENCODE('TEST')")
expr = self.parse_one("TO_BINARY('SEVMUA==', 'BASE64')", dialect="snowflake")
annotated = annotate_types(expr, dialect="snowflake")
self.assertEqual(annotated.sql("duckdb"), "FROM_BASE64('SEVMUA==')")
self.validate_all(
"[0, 1, 2]",
read={
@ -1290,6 +1298,38 @@ class TestDuckDB(Validator):
self.validate_identity("SELECT 20_000 AS literal")
self.validate_identity("SELECT 1_2E+1_0::FLOAT", "SELECT CAST(1_2E+1_0 AS REAL)")
# Test BITMAP_BUCKET_NUMBER transpilation from Snowflake to DuckDB
self.validate_all(
"CASE WHEN 2500 > 0 THEN ((2500 - 1) // 32768) + 1 ELSE 2500 // 32768 END",
read={
"snowflake": "BITMAP_BUCKET_NUMBER(2500)",
},
)
self.validate_all(
"CASE WHEN 32768 > 0 THEN ((32768 - 1) // 32768) + 1 ELSE 32768 // 32768 END",
read={
"snowflake": "BITMAP_BUCKET_NUMBER(32768)",
},
)
self.validate_all(
"CASE WHEN 32769 > 0 THEN ((32769 - 1) // 32768) + 1 ELSE 32769 // 32768 END",
read={
"snowflake": "BITMAP_BUCKET_NUMBER(32769)",
},
)
self.validate_all(
"CASE WHEN -100 > 0 THEN ((-100 - 1) // 32768) + 1 ELSE -100 // 32768 END",
read={
"snowflake": "BITMAP_BUCKET_NUMBER(-100)",
},
)
self.validate_all(
"CASE WHEN NULL > 0 THEN ((NULL - 1) // 32768) + 1 ELSE NULL // 32768 END",
read={
"snowflake": "BITMAP_BUCKET_NUMBER(NULL)",
},
)
def test_array_index(self):
with self.assertLogs(helper_logger) as cm:
self.validate_all(
@ -1356,11 +1396,17 @@ class TestDuckDB(Validator):
read={"bigquery": "SELECT DATE(DATETIME '2016-12-25 23:59:59')"},
)
self.validate_all(
"SELECT STRPTIME(STRFTIME(CAST(CAST('2016-12-25' AS TIMESTAMPTZ) AS DATE), '%d/%m/%Y') || ' ' || 'America/Los_Angeles', '%d/%m/%Y %Z')",
"SELECT CAST(CAST(CAST('2016-12-25' AS TIMESTAMPTZ) AS TIMESTAMP) AT TIME ZONE 'UTC' AT TIME ZONE 'America/Los_Angeles' AS DATE)",
read={
"bigquery": "SELECT DATE(TIMESTAMP '2016-12-25', 'America/Los_Angeles')",
},
)
self.validate_all(
"SELECT CAST(CAST('2024-01-15 23:30:00' AS TIMESTAMP) AT TIME ZONE 'UTC' AT TIME ZONE 'Europe/Berlin' AS DATE)",
read={
"bigquery": "SELECT DATE('2024-01-15 23:30:00', 'Europe/Berlin')",
},
)
self.validate_all(
"SELECT CAST(CAST(STRPTIME('05/06/2020', '%m/%d/%Y') AS DATE) AS DATE)",
read={"bigquery": "SELECT DATE(PARSE_DATE('%m/%d/%Y', '05/06/2020'))"},
@ -2092,3 +2138,54 @@ class TestDuckDB(Validator):
self.validate_identity(
f"CREATE {keyword} ifelse(a, b, c) AS CASE WHEN a THEN b ELSE c END"
)
def test_bitwise_agg(self):
self.validate_all(
"SELECT BIT_OR(int_value) FROM t",
read={
"snowflake": "SELECT BITOR_AGG(int_value) FROM t",
"duckdb": "SELECT BIT_OR(int_value) FROM t",
},
)
self.validate_all(
"SELECT BIT_AND(int_value) FROM t",
read={
"snowflake": "SELECT BITAND_AGG(int_value) FROM t",
"duckdb": "SELECT BIT_AND(int_value) FROM t",
},
)
self.validate_all(
"SELECT BIT_XOR(int_value) FROM t",
read={
"snowflake": "SELECT BITXOR_AGG(int_value) FROM t",
"duckdb": "SELECT BIT_XOR(int_value) FROM t",
},
)
self.validate_all(
"SELECT BIT_OR(CAST(val AS FLOAT)) FROM t",
write={
"duckdb": "SELECT BIT_OR(CAST(ROUND(CAST(val AS REAL)) AS INT)) FROM t",
"snowflake": "SELECT BITORAGG(CAST(val AS FLOAT)) FROM t",
},
)
self.validate_all(
"SELECT BIT_AND(CAST(val AS DOUBLE)) FROM t",
write={
"duckdb": "SELECT BIT_AND(CAST(ROUND(CAST(val AS DOUBLE)) AS INT)) FROM t",
"snowflake": "SELECT BITANDAGG(CAST(val AS DOUBLE)) FROM t",
},
)
self.validate_all(
"SELECT BIT_OR(CAST(val AS DECIMAL(10, 2))) FROM t",
write={
"duckdb": "SELECT BIT_OR(CAST(CAST(val AS DECIMAL(10, 2)) AS INT)) FROM t",
"snowflake": "SELECT BITORAGG(CAST(val AS DECIMAL(10, 2))) FROM t",
},
)
self.validate_all(
"SELECT BIT_XOR(CAST(val AS DECIMAL)) FROM t",
write={
"duckdb": "SELECT BIT_XOR(CAST(CAST(val AS DECIMAL(18, 3)) AS INT)) FROM t",
"snowflake": "SELECT BITXORAGG(CAST(val AS DECIMAL(18, 3))) FROM t",
},
)

21
tests/dialects/test_exasol.py
View file

@ -10,6 +10,9 @@ class TestExasol(Validator):
"SELECT 1 AS [x]",
'SELECT 1 AS "x"',
)
self.validate_identity("SYSTIMESTAMP", "SYSTIMESTAMP()")
self.validate_identity("SELECT SYSTIMESTAMP()")
self.validate_identity("SELECT SYSTIMESTAMP(6)")
def test_qualify_unscoped_star(self):
self.validate_all(
@ -336,10 +339,10 @@ class TestExasol(Validator):
"SELECT TO_CHAR(CAST('1999-12-31' AS DATE)) AS TO_CHAR",
write={
"exasol": "SELECT TO_CHAR(CAST('1999-12-31' AS DATE)) AS TO_CHAR",
"redshift": "SELECT TO_CHAR(CAST('1999-12-31' AS DATE)) AS TO_CHAR",
"presto": "SELECT DATE_FORMAT(CAST('1999-12-31' AS DATE)) AS TO_CHAR",
"oracle": "SELECT TO_CHAR(CAST('1999-12-31' AS DATE)) AS TO_CHAR",
"postgres": "SELECT TO_CHAR(CAST('1999-12-31' AS DATE)) AS TO_CHAR",
"redshift": "SELECT CAST(CAST('1999-12-31' AS DATE) AS VARCHAR(MAX)) AS TO_CHAR",
"postgres": "SELECT CAST(CAST('1999-12-31' AS DATE) AS TEXT) AS TO_CHAR",
},
read={
"exasol": "SELECT TO_CHAR(DATE '1999-12-31') AS TO_CHAR",
@ -423,6 +426,16 @@ class TestExasol(Validator):
f"SELECT TO_CHAR(CAST('2024-07-08 13:45:00' AS TIMESTAMP), '{fmt}') AS {alias}"
)
self.validate_all(
"SELECT TO_CHAR(CAST('2024-07-08 13:45:00' AS TIMESTAMP), 'DY')",
write={
"exasol": "SELECT TO_CHAR(CAST('2024-07-08 13:45:00' AS TIMESTAMP), 'DY')",
"oracle": "SELECT TO_CHAR(CAST('2024-07-08 13:45:00' AS TIMESTAMP), 'DY')",
"postgres": "SELECT TO_CHAR(CAST('2024-07-08 13:45:00' AS TIMESTAMP), 'TMDy')",
"databricks": "SELECT DATE_FORMAT(CAST('2024-07-08 13:45:00' AS TIMESTAMP), 'EEE')",
},
)
self.validate_all(
"TO_DATE(x, 'YYYY-MM-DD')",
write={
@ -481,7 +494,7 @@ class TestExasol(Validator):
)
units = ["MM", "QUARTER", "WEEK", "MINUTE", "YEAR"]
for unit in units:
with self.subTest(f"Testing TO_CHAR with format '{unit}'"):
with self.subTest(f"Testing DATE_TRUNC with format '{unit}'"):
self.validate_all(
f"SELECT TRUNC(CAST('2006-12-31' AS DATE), '{unit}') AS TRUNC",
write={
@ -705,6 +718,8 @@ class TestExasol(Validator):
"SELECT name, age, IF age < 18 THEN 'underaged' ELSE 'adult' ENDIF AS LEGALITY FROM persons"
)
self.validate_identity("SELECT HASHTYPE_MD5(a, b, c, d)")
def test_odbc_date_literals(self):
self.validate_identity("SELECT {d'2024-01-01'}", "SELECT TO_DATE('2024-01-01')")
self.validate_identity(

11
tests/dialects/test_mysql.py
View file

@ -193,6 +193,9 @@ class TestMySQL(Validator):
self.validate_identity("ALTER TABLE t ALTER INDEX i VISIBLE")
self.validate_identity("ALTER TABLE t ALTER COLUMN c SET INVISIBLE")
self.validate_identity("ALTER TABLE t ALTER COLUMN c SET VISIBLE")
self.validate_identity(
"UPDATE foo JOIN bar ON TRUE SET foo.a = bar.a WHERE foo.id = bar.id"
)
def test_identity(self):
self.validate_identity("SELECT HIGH_PRIORITY STRAIGHT_JOIN SQL_CALC_FOUND_ROWS * FROM t")
@ -332,6 +335,7 @@ class TestMySQL(Validator):
)
self.validate_identity("SELECT SUBSTR(1 FROM 2 FOR 3)", "SELECT SUBSTRING(1, 2, 3)")
self.validate_identity("SELECT ELT(2, 'foo', 'bar', 'baz') AS Result")
self.validate_identity("SELECT CHARSET(CHAR(100 USING utf8))")
def test_types(self):
for char_type in MySQL.Generator.CHAR_CAST_MAPPING:
@ -482,7 +486,7 @@ class TestMySQL(Validator):
"clickhouse": UnsupportedError,
"databricks": "SELECT X'CC'",
"drill": "SELECT 204",
"duckdb": "SELECT CAST(HEX(FROM_HEX('CC')) AS VARBINARY)",
"duckdb": "SELECT UNHEX('CC')",
"hive": "SELECT 204",
"mysql": "SELECT x'CC'",
"oracle": "SELECT 204",
@ -503,7 +507,7 @@ class TestMySQL(Validator):
"clickhouse": UnsupportedError,
"databricks": "SELECT X'0000CC'",
"drill": "SELECT 204",
"duckdb": "SELECT CAST(HEX(FROM_HEX('0000CC')) AS VARBINARY)",
"duckdb": "SELECT UNHEX('0000CC')",
"hive": "SELECT 204",
"mysql": "SELECT x'0000CC'",
"oracle": "SELECT 204",
@ -573,6 +577,9 @@ class TestMySQL(Validator):
"mysql": "CAST(x AS CHAR CHARACTER SET latin1)",
},
)
self.validate_identity(
"CONVERT('a' USING binary)", "CAST('a' AS CHAR CHARACTER SET binary)"
)
def test_match_against(self):
self.validate_all(

71
tests/dialects/test_oracle.py
View file

@ -1,4 +1,4 @@
from sqlglot import exp, UnsupportedError, ParseError, parse_one
from sqlglot import exp, UnsupportedError, ParseError, parse_one, parse
from tests.dialects.test_dialect import Validator
from sqlglot.optimizer.qualify import qualify
@ -17,6 +17,7 @@ class TestOracle(Validator):
)
self.parse_one("ALTER TABLE tbl_name DROP FOREIGN KEY fk_symbol").assert_is(exp.Alter)
self.validate_identity("XMLELEMENT(EVALNAME foo + bar)")
self.validate_identity("SELECT BITMAP_BUCKET_NUMBER(32769)")
self.validate_identity("SELECT BITMAP_CONSTRUCT_AGG(value)")
self.validate_identity("DBMS_RANDOM.NORMAL")
@ -28,6 +29,8 @@ class TestOracle(Validator):
self.validate_identity("CREATE PRIVATE TEMPORARY TABLE t AS SELECT * FROM orders")
self.validate_identity("REGEXP_REPLACE('source', 'search')")
self.validate_identity("TIMESTAMP(3) WITH TIME ZONE")
self.validate_identity("SYSTIMESTAMP").assert_is(exp.Systimestamp)
self.validate_identity("SELECT SYSTIMESTAMP AT TIME ZONE 'UTC'")
self.validate_identity("CURRENT_TIMESTAMP(precision)")
self.validate_identity("ALTER TABLE tbl_name DROP FOREIGN KEY fk_symbol")
self.validate_identity("ALTER TABLE Payments ADD Stock NUMBER NOT NULL")
@ -53,6 +56,12 @@ class TestOracle(Validator):
self.validate_identity("SELECT * FROM V$SESSION")
self.validate_identity("SELECT TO_DATE('January 15, 1989, 11:00 A.M.')")
self.validate_identity("SELECT INSTR(haystack, needle)")
self.validate_identity(
"SELECT (TIMESTAMP '2025-12-30 20:00:00' - TIMESTAMP '2025-12-29 14:30:00') DAY TO SECOND",
"SELECT (TO_TIMESTAMP('2025-12-30 20:00:00', 'YYYY-MM-DD HH24:MI:SS.FF6') - TO_TIMESTAMP('2025-12-29 14:30:00', 'YYYY-MM-DD HH24:MI:SS.FF6')) DAY TO SECOND",
)
self.validate_identity("SELECT (SYSTIMESTAMP - order_date) DAY(9) TO SECOND FROM orders")
self.validate_identity("SELECT (SYSTIMESTAMP - order_date) DAY(9) TO SECOND(3) FROM orders")
self.validate_identity(
"SELECT * FROM consumer LEFT JOIN groceries ON consumer.groceries_id = consumer.id PIVOT(MAX(type_id) FOR consumer_type IN (1, 2, 3, 4))"
)
@ -89,6 +98,10 @@ class TestOracle(Validator):
self.validate_identity(
"SELECT MIN(column_name) KEEP (DENSE_RANK FIRST ORDER BY column_name DESC) FROM table_name"
)
self.validate_identity(
'XMLELEMENT("ImageID", image.id)',
'XMLELEMENT(NAME "ImageID", image.id)',
)
self.validate_identity(
"SELECT CAST('January 15, 1989, 11:00 A.M.' AS DATE DEFAULT NULL ON CONVERSION ERROR, 'Month dd, YYYY, HH:MI A.M.') FROM DUAL",
"SELECT TO_DATE('January 15, 1989, 11:00 A.M.', 'Month dd, YYYY, HH12:MI A.M.') FROM DUAL",
@ -232,13 +245,6 @@ class TestOracle(Validator):
"teradata": "TO_NUMBER(x, fmt)",
},
)
self.validate_all(
"SELECT TO_CHAR(TIMESTAMP '1999-12-01 10:00:00')",
write={
"oracle": "SELECT TO_CHAR(CAST('1999-12-01 10:00:00' AS TIMESTAMP))",
"postgres": "SELECT TO_CHAR(CAST('1999-12-01 10:00:00' AS TIMESTAMP))",
},
)
self.validate_all(
"SELECT CAST(NULL AS VARCHAR2(2328 CHAR)) AS COL1",
write={
@ -813,3 +819,52 @@ CONNECT BY PRIOR employee_id = manager_id AND LEVEL <= 4"""
qualified.sql(dialect="oracle"),
'WITH "T" AS (SELECT 1 AS "COL") SELECT "T"."COL" AS "COL", ROWID AS "ROWID" FROM "T" "T" WHERE ROWNUM = 1',
)
def test_chr(self):
self.validate_identity("SELECT CHR(187 USING NCHAR_CS)")
self.validate_identity("SELECT CHR(187)")
def test_full_procedure(self):
sql = """
CREATE OR REPLACE PROCEDURE query_emp(
p_id IN VARCHAR2,
p_name OUT VARCHAR2,
p_salary OUT NUMBER
) AS
BEGIN
SELECT last_name, salary
INTO p_name, p_salary
FROM employees
WHERE employee_id = p_id;
END;
"""
expected_sqls = [
"CREATE OR REPLACE PROCEDURE query_emp(p_id IN VARCHAR2, p_name OUT VARCHAR2, p_salary OUT NUMBER) AS BEGIN SELECT last_name, salary INTO p_name, p_salary FROM employees WHERE employee_id = p_id",
"END",
]
for expr, expected_sql in zip(parse(sql, read="oracle"), expected_sqls):
self.assertEqual(expr.sql(dialect="oracle"), expected_sql)
sql = """
CREATE OR REPLACE PROCEDURE test_proc (
a NUMBER,
b IN NUMBER,
c IN OUT NUMBER,
d OUT NUMBER
) AS
BEGIN
c := c + a + b;
d := 42 + c;
END;
"""
expected_sqls = [
"CREATE OR REPLACE PROCEDURE test_proc(a NUMBER, b IN NUMBER, c IN OUT NUMBER, d OUT NUMBER) AS BEGIN c := c + a + b",
"d := 42 + c",
"END",
]
for expr, expected_sql in zip(parse(sql, read="oracle"), expected_sqls):
self.assertEqual(expr.sql(dialect="oracle"), expected_sql)

91
tests/dialects/test_postgres.py
View file

@ -22,6 +22,7 @@ class TestPostgres(Validator):
expected_sql = "ARRAY[\n x" + (",\n x" * 27) + "\n]"
self.validate_identity(sql, expected_sql, pretty=True)
self.validate_identity("SELECT GET_BIT(CAST(44 AS BIT(10)), 6)")
self.validate_identity("SELECT * FROM t GROUP BY ROLLUP (a || '^' || b)")
self.validate_identity("SELECT COSH(1.5)")
self.validate_identity("SELECT EXP(1)")
@ -993,6 +994,46 @@ FROM json_data, field_ids""",
},
)
self.validate_all(
"SELECT TO_CHAR(foo, bar)",
read={
"redshift": "SELECT TO_CHAR(foo, bar)",
},
write={
"postgres": "SELECT TO_CHAR(foo, bar)",
"redshift": "SELECT TO_CHAR(foo, bar)",
},
)
self.validate_all(
"CREATE TABLE table1 (a INT, b INT, PRIMARY KEY (a))",
read={
"sqlite": "CREATE TABLE table1 (a INT, b INT, PRIMARY KEY (a))",
"postgres": "CREATE TABLE table1 (a INT, b INT, PRIMARY KEY (a))",
},
)
self.validate_identity("SELECT NUMRANGE(1.1, 2.2) -|- NUMRANGE(2.2, 3.3)")
self.validate_identity(
"SELECT SLOPE(point '(4,4)', point '(0,0)')",
"SELECT SLOPE(CAST('(4,4)' AS POINT), CAST('(0,0)' AS POINT))",
)
width_bucket = self.validate_identity("WIDTH_BUCKET(10, ARRAY[5, 15])")
self.assertIsNotNone(width_bucket.args.get("threshold"))
width_bucket = self.validate_identity("WIDTH_BUCKET(10, 5, 15, 25)")
self.assertIsNone(width_bucket.args.get("threshold"))
self.validate_all(
"UPDATE foo SET a = bar.a, b = bar.b FROM bar WHERE foo.id = bar.id",
write={
"postgres": "UPDATE foo SET a = bar.a, b = bar.b FROM bar WHERE foo.id = bar.id",
"doris": "UPDATE foo SET a = bar.a, b = bar.b FROM bar WHERE foo.id = bar.id",
"starrocks": "UPDATE foo SET a = bar.a, b = bar.b FROM bar WHERE foo.id = bar.id",
"mysql": "UPDATE foo JOIN bar ON TRUE SET foo.a = bar.a, foo.b = bar.b WHERE foo.id = bar.id",
"singlestore": "UPDATE foo JOIN bar ON TRUE SET foo.a = bar.a, foo.b = bar.b WHERE foo.id = bar.id",
},
)
def test_ddl(self):
# Checks that user-defined types are parsed into DataType instead of Identifier
self.parse_one("CREATE TABLE t (a udt)").this.expressions[0].args["kind"].assert_is(
@ -1044,6 +1085,9 @@ FROM json_data, field_ids""",
self.validate_identity("ALTER TABLE t1 SET ACCESS METHOD method")
self.validate_identity("ALTER TABLE t1 SET TABLESPACE tablespace")
self.validate_identity("ALTER TABLE t1 SET (fillfactor = 5, autovacuum_enabled = TRUE)")
self.validate_identity(
"INSERT INTO book (isbn, title) VALUES ($1, $2) ON CONFLICT(isbn) WHERE deleted_at IS NULL DO UPDATE SET title = EXCLUDED.title RETURNING id, isbn"
)
self.validate_identity(
"INSERT INTO newtable AS t(a, b, c) VALUES (1, 2, 3) ON CONFLICT(c) DO UPDATE SET a = t.a + 1 WHERE t.a < 1"
)
@ -1229,6 +1273,9 @@ FROM json_data, field_ids""",
self.validate_identity("CREATE TABLE tbl (col UUID UNIQUE DEFAULT GEN_RANDOM_UUID())")
self.validate_identity("CREATE TABLE tbl (col UUID, UNIQUE NULLS NOT DISTINCT (col))")
self.validate_identity("CREATE TABLE tbl (col_a INT GENERATED ALWAYS AS (1 + 2) STORED)")
self.validate_identity(
"CREATE TABLE tbl (col_a INTERVAL GENERATED ALWAYS AS (a - b) STORED)"
)
self.validate_identity("CREATE INDEX CONCURRENTLY ix_table_id ON tbl USING btree(id)")
self.validate_identity(
@ -1315,20 +1362,6 @@ FROM json_data, field_ids""",
],
)
def test_operator(self):
expr = self.parse_one("1 OPERATOR(+) 2 OPERATOR(*) 3")
expr.left.assert_is(exp.Operator)
expr.left.left.assert_is(exp.Literal)
expr.left.right.assert_is(exp.Literal)
expr.right.assert_is(exp.Literal)
self.assertEqual(expr.sql(dialect="postgres"), "1 OPERATOR(+) 2 OPERATOR(*) 3")
self.validate_identity("SELECT operator FROM t")
self.validate_identity("SELECT 1 OPERATOR(+) 2")
self.validate_identity("SELECT 1 OPERATOR(+) /* foo */ 2")
self.validate_identity("SELECT 1 OPERATOR(pg_catalog.+) 2")
def test_bool_or(self):
self.validate_identity(
"SELECT a, LOGICAL_OR(b) FROM table GROUP BY a",
@ -1383,6 +1416,36 @@ FROM json_data, field_ids""",
},
)
def test_corr(self):
self.validate_all(
"SELECT CORR(a, b)",
write={
"duckdb": "SELECT CORR(a, b)",
"postgres": "SELECT CORR(a, b)",
},
)
self.validate_all(
"SELECT CORR(a, b) OVER (PARTITION BY c)",
write={
"duckdb": "SELECT CORR(a, b) OVER (PARTITION BY c)",
"postgres": "SELECT CORR(a, b) OVER (PARTITION BY c)",
},
)
self.validate_all(
"SELECT CORR(a, b) FILTER(WHERE c > 0)",
write={
"duckdb": "SELECT CORR(a, b) FILTER(WHERE c > 0)",
"postgres": "SELECT CORR(a, b) FILTER(WHERE c > 0)",
},
)
self.validate_all(
"SELECT CORR(a, b) FILTER(WHERE c > 0) OVER (PARTITION BY d)",
write={
"duckdb": "SELECT CORR(a, b) FILTER(WHERE c > 0) OVER (PARTITION BY d)",
"postgres": "SELECT CORR(a, b) FILTER(WHERE c > 0) OVER (PARTITION BY d)",
},
)
def test_regexp_binary(self):
"""See https://github.com/tobymao/sqlglot/pull/2404 for details."""
self.assertIsInstance(self.parse_one("'thomas' ~ '.*thomas.*'"), exp.Binary)

12
tests/dialects/test_presto.py
View file

@ -495,6 +495,18 @@ class TestPresto(Validator):
},
)
self.validate_all(
"SELECT WEEK_OF_YEAR(y)",
read={
"presto": "SELECT WEEK(y)",
},
write={
"spark": "SELECT WEEKOFYEAR(y)",
"presto": "SELECT WEEK_OF_YEAR(y)",
"trino": "SELECT WEEK_OF_YEAR(y)",
},
)
def test_ddl(self):
self.validate_all(
"CREATE TABLE test WITH (FORMAT = 'PARQUET') AS SELECT 1",

1
tests/dialects/test_redshift.py
View file

@ -330,6 +330,7 @@ class TestRedshift(Validator):
)
def test_identity(self):
self.validate_identity("SELECT GETBIT(FROM_HEX('4d'), 2)")
self.validate_identity("SELECT EXP(1)")
self.validate_identity("ALTER TABLE table_name ALTER COLUMN bla TYPE VARCHAR")
self.validate_identity("SELECT CAST(value AS FLOAT(8))")

4
tests/dialects/test_singlestore.py
View file

@ -20,6 +20,10 @@ class TestSingleStore(Validator):
self.validate_identity("SELECT 1")
self.validate_identity("SELECT * FROM `users` ORDER BY ALL")
self.validate_identity("SELECT ELT(2, 'foo', 'bar', 'baz')")
self.validate_identity("SELECT CHARSET(CHAR(100 USING utf8))")
self.validate_identity("SELECT TO_JSON(ROW(1, 2) :> RECORD(a INT, b INT))")
self.validate_identity("JSON_KEYS(json_doc, 'a', 'b', 'c', 2)")
def test_byte_strings(self):
self.validate_identity("SELECT e'text'")

893
tests/dialects/test_snowflake.py
View file

File diff suppressed because it is too large Load diff

12
tests/dialects/test_spark.py
View file

@ -649,6 +649,17 @@ TBLPROPERTIES (
},
)
self.validate_all(
"SELECT * FROM quarterly_sales PIVOT(SUM(amount) AS amount, 'dummy' AS bar FOR quarter IN ('2023_Q1'))",
read={
"spark": "SELECT * FROM quarterly_sales PIVOT(SUM(amount) amount, 'dummy' bar FOR quarter IN ('2023_Q1'))",
"databricks": "SELECT * FROM quarterly_sales PIVOT(SUM(amount) amount, 'dummy' bar FOR quarter IN ('2023_Q1'))",
},
write={
"databricks": "SELECT * FROM quarterly_sales PIVOT(SUM(amount) AS amount, 'dummy' AS bar FOR quarter IN ('2023_Q1'))",
},
)
for data_type in (
"BOOLEAN",
"DATE",
@ -928,6 +939,7 @@ TBLPROPERTIES (
)
self.validate_identity("BITMAP_OR_AGG(x)")
self.validate_identity("SELECT ELT(2, 'foo', 'bar', 'baz') AS Result")
self.validate_identity("SELECT MAKE_INTERVAL(100, 11, 12, 13, 14, 14, 15)")
def test_bool_or(self):
self.validate_all(

2
tests/dialects/test_teradata.py
View file

@ -70,7 +70,7 @@ class TestTeradata(Validator):
"UPDATE A FROM schema.tableA AS A, (SELECT col1 FROM schema.tableA GROUP BY col1) AS B SET col2 = '' WHERE A.col1 = B.col1",
write={
"teradata": "UPDATE A FROM schema.tableA AS A, (SELECT col1 FROM schema.tableA GROUP BY col1) AS B SET col2 = '' WHERE A.col1 = B.col1",
"mysql": "UPDATE A SET col2 = '' FROM `schema`.tableA AS A, (SELECT col1 FROM `schema`.tableA GROUP BY col1) AS B WHERE A.col1 = B.col1",
"mysql": "UPDATE A JOIN `schema`.tableA AS A ON TRUE JOIN (SELECT col1 FROM `schema`.tableA GROUP BY col1) AS B ON TRUE SET A.col2 = '' WHERE A.col1 = B.col1",
},
)

3
tests/fixtures/identity.sql vendored
View file

@ -951,4 +951,5 @@ SELECT CURRENT_DATABASE()
SELECT CURRENT_SCHEMAS(arg_bool)
SELECT UNIFORM(1, 10, 5)
SELECT UNIFORM(1, 10)
SELECT CURRENT_TIMEZONE()
SELECT CURRENT_TIMEZONE()
SELECT NUMRANGE(1.1, 2.2) -|- NUMRANGE(2.2, 3.3)

312
tests/fixtures/optimizer/annotate_functions.sql vendored
View file

@ -22,6 +22,9 @@ DOUBLE;
CURRENT_TIME();
TIME;
LOCALTIME();
TIME;
TIME_ADD(CAST('09:05:03' AS TIME), INTERVAL 2 HOUR);
TIME;
@ -320,6 +323,47 @@ INTERVAL;
COALESCE(tbl.bin_col, tbl.str_col);
BINARY;
# dialect: spark, databricks
LOCALTIMESTAMP();
TIMESTAMPNTZ;
# dialect: hive, spark2, spark, databricks
ENCODE(tbl.str_col, tbl.str_col);
BINARY;
# dialect: hive, spark2, spark, databricks
ENCODE(tbl.bin_col, tbl.bin_col);
BINARY;
# dialect: spark, databricks
CURRENT_TIMEZONE();
STRING;
# dialect: hive, spark2, spark, databricks
UNIX_TIMESTAMP();
BIGINT;
# dialect: spark2, spark, databricks
ATAN2(tbl.int_col, tbl.int_col);
DOUBLE;
# dialect: spark2, spark, databricks
ATAN2(tbl.double_col, tbl.double_col);
DOUBLE;
# dialect: spark2, spark, databricks
ATAN2(tbl.double_col, tbl.int_col);
DOUBLE;
# dialect: hive, spark2, spark, databricks
TANH(tbl.double_col);
DOUBLE;
# dialect: hive, spark2, spark, databricks
TANH(tbl.int_col);
DOUBLE;
--------------------------------------
-- BigQuery
--------------------------------------
@ -1676,6 +1720,14 @@ DATETIME;
IF(TRUE, '00:00:00', TIME '00:01:00');
TIME;
# dialect: bigquery
IF(TRUE, 1, CAST(2.5 AS BIGNUMERIC));
BIGNUMERIC;
# dialect: bigquery
IF(TRUE, 1.5, 2.5);
FLOAT64;
# dialect: bigquery
IF(TRUE, '2010-01-01 00:00:00', TIMESTAMP '2020-02-02 00:00:00');
TIMESTAMP;
@ -1704,6 +1756,10 @@ STRING;
ANY_VALUE(c2::STRING HAVING MAX c1::INT64);
STRING;
# dialect: bigquery
r'a';
STRING;
--------------------------------------
-- Snowflake
--------------------------------------
@ -1724,6 +1780,10 @@ DATE;
ADD_MONTHS(tbl.timestamp_col, -1);
TIMESTAMP;
# dialect: snowflake
ARRAY_CONSTRUCT();
ARRAY;
# dialect: snowflake
ARRAY_CONSTRUCT_COMPACT();
ARRAY;
@ -1732,6 +1792,22 @@ ARRAY;
ARRAY_CONSTRUCT_COMPACT(1, null, 2);
ARRAY;
# dialect: snowflake
ARRAY_APPEND([1, 2, 3], 4);
ARRAY;
# dialect: snowflake
ARRAY_CAT([1, 2], [3, 4]);
ARRAY;
# dialect: snowflake
ARRAY_PREPEND([2, 3, 4], 1);
ARRAY;
# dialect: snowflake
ARRAY_REMOVE([1, 2, 3], 2);
ARRAY;
# dialect: snowflake
ASIN(tbl.double_col);
DOUBLE;
@ -1756,6 +1832,22 @@ DOUBLE;
CBRT(tbl.double_col);
DOUBLE;
# dialect: snowflake
COVAR_POP(tbl.double_col, tbl.double_col);
DOUBLE;
# dialect: snowflake
COVAR_SAMP(tbl.double_col, tbl.double_col);
DOUBLE;
# dialect: snowflake
COVAR_POP(tbl.double_col, tbl.double_col) OVER (PARTITION BY 1);
DOUBLE;
# dialect: snowflake
COVAR_SAMP(tbl.double_col, tbl.double_col) OVER (PARTITION BY 1);
DOUBLE;
# dialect: snowflake
AI_AGG('foo', 'bar');
VARCHAR;
@ -2800,6 +2892,38 @@ VARCHAR;
INSERT(tbl.bin_col, 1, 2, tbl.bin_col);
BINARY;
# dialect: snowflake
KURTOSIS(tbl.double_col);
DOUBLE;
# dialect: snowflake
KURTOSIS(tbl.double_col) OVER (PARTITION BY 1);
DOUBLE;
# dialect: snowflake
KURTOSIS(tbl.float_col);
DOUBLE;
# dialect: snowflake
KURTOSIS(tbl.float_col) OVER (PARTITION BY 1);
DOUBLE;
# dialect: snowflake
KURTOSIS(tbl.int_col);
NUMBER(38, 12);
# dialect: snowflake
KURTOSIS(tbl.int_col) OVER (PARTITION BY 1);
NUMBER(38, 12);
# dialect: snowflake
KURTOSIS(tbl.decfloat_col);
DECFLOAT;
# dialect: snowflake
KURTOSIS(tbl.decfloat_col) OVER (PARTITION BY 1);
DECFLOAT;
# dialect: snowflake
LEAST(x::DECIMAL(18, 2));
DECIMAL(18, 2);
@ -2848,6 +2972,18 @@ INT;
LENGTH(tbl.bin_col);
INT;
# dialect: snowflake
LOCALTIMESTAMP;
TIMESTAMPLTZ;
# dialect: snowflake
LOCALTIMESTAMP();
TIMESTAMPLTZ;
# dialect: snowflake
LOCALTIMESTAMP(3);
TIMESTAMPLTZ;
# dialect: snowflake
OCTET_LENGTH(tbl.str_col);
INT;
@ -2952,6 +3088,34 @@ VARCHAR;
LTRIM(NULL);
VARCHAR;
# dialect: snowflake
MAP_CAT(CAST(col AS MAP(VARCHAR, VARCHAR)), CAST(col AS MAP(VARCHAR, VARCHAR)));
MAP;
# dialect: snowflake
MAP_CONTAINS_KEY('k1', CAST(col AS MAP(VARCHAR, VARCHAR)));
BOOLEAN;
# dialect: snowflake
MAP_DELETE(CAST(col AS MAP(VARCHAR, VARCHAR)), 'b');
MAP;
# dialect: snowflake
MAP_INSERT(CAST(col AS MAP(VARCHAR, VARCHAR)), 'b', '2');
MAP;
# dialect: snowflake
MAP_KEYS(CAST(col AS MAP(VARCHAR, VARCHAR)));
ARRAY;
# dialect: snowflake
MAP_PICK(CAST(col AS MAP(VARCHAR, VARCHAR)), 'a', 'c');
MAP;
# dialect: snowflake
MAP_SIZE(CAST(col AS MAP(VARCHAR, VARCHAR)));
INT;
# dialect: snowflake
MINUTE(CAST('08:50:57' AS TIME));
INT;
@ -3732,6 +3896,14 @@ TIME;
TRY_TO_TIME('12:30:00', 'AUTO');
TIME;
# dialect: snowflake
TO_TIME('093000', 'HH24MISS');
TIME;
# dialect: snowflake
TRY_TO_TIME('093000', 'HH24MISS');
TIME;
# dialect: snowflake
TO_TIMESTAMP('2024-01-15 12:30:00');
TIMESTAMP;
@ -3772,6 +3944,30 @@ TIMESTAMP;
TRY_TO_TIMESTAMP('2024-01-15 12:30:00', 'AUTO');
TIMESTAMP;
# dialect: snowflake
TRY_TO_TIMESTAMP_LTZ('2024-01-15 12:30:00');
TIMESTAMPLTZ;
# dialect: snowflake
TRY_TO_TIMESTAMP_LTZ('2024-01-15 12:30:00', 'AUTO');
TIMESTAMPLTZ;
# dialect: snowflake
TRY_TO_TIMESTAMP_NTZ('2024-01-15 12:30:00');
TIMESTAMPNTZ;
# dialect: snowflake
TRY_TO_TIMESTAMP_NTZ('2024-01-15 12:30:00', 'AUTO');
TIMESTAMPNTZ;
# dialect: snowflake
TRY_TO_TIMESTAMP_TZ('2024-01-15 12:30:00');
TIMESTAMPTZ;
# dialect: snowflake
TRY_TO_TIMESTAMP_TZ('2024-01-15 12:30:00', 'AUTO');
TIMESTAMPTZ;
# dialect: snowflake
ABS(CAST(-123.456 AS DECFLOAT));
DECFLOAT;
@ -4052,6 +4248,30 @@ BOOLEAN;
SEARCH_IP(col, '192.168.0.0');
BOOLEAN;
# dialect: snowflake
STDDEV(tbl.double_col);
DOUBLE;
# dialect: snowflake
STDDEV(tbl.double_col) OVER (PARTITION BY 1);
DOUBLE;
# dialect: snowflake
STDDEV_POP(tbl.double_col);
DOUBLE;
# dialect: snowflake
STDDEV_POP(tbl.double_col) OVER (PARTITION BY 1);
DOUBLE;
# dialect: snowflake
STDDEV_SAMP(tbl.double_col);
DOUBLE;
# dialect: snowflake
STDDEV_SAMP(tbl.double_col) OVER (PARTITION BY 1);
DOUBLE;
# dialect: snowflake
STRTOK_TO_ARRAY('a,b,c', ',');
ARRAY;
@ -4648,6 +4868,86 @@ NUMBER(38, 15);
VARIANCE_POP(1::NUMBER(30, 5));
NUMBER(38, 12);
# dialect: snowflake
ENCRYPT(tbl.str_col, 'passphrase');
BINARY;
# dialect: snowflake
ENCRYPT(tbl.str_col, 'passphrase', 'aad');
BINARY;
# dialect: snowflake
ENCRYPT(tbl.str_col, 'passphrase', 'aad', 'AES-GCM');
BINARY;
# dialect: snowflake
ENCRYPT_RAW(tbl.str_col, tbl.key_col, tbl.iv_col);
BINARY;
# dialect: snowflake
ENCRYPT_RAW(tbl.str_col, tbl.key_col, tbl.iv_col, tbl.aad_col);
BINARY;
# dialect: snowflake
ENCRYPT_RAW(tbl.str_col, tbl.key_col, tbl.iv_col, tbl.aad_col, 'AES-GCM');
BINARY;
# dialect: snowflake
DECRYPT(tbl.encrypted_col, 'passphrase');
BINARY;
# dialect: snowflake
DECRYPT(tbl.encrypted_col, 'passphrase', 'aad');
BINARY;
# dialect: snowflake
DECRYPT(tbl.encrypted_col, 'passphrase', 'aad', 'AES-GCM');
BINARY;
# dialect: snowflake
DECRYPT_RAW(tbl.encrypted_col, tbl.key_col, tbl.iv_col);
BINARY;
# dialect: snowflake
DECRYPT_RAW(tbl.encrypted_col, tbl.key_col, tbl.iv_col, tbl.aad_col);
BINARY;
# dialect: snowflake
DECRYPT_RAW(tbl.encrypted_col, tbl.key_col, tbl.iv_col, tbl.aad_col, 'AES-GCM');
BINARY;
# dialect: snowflake
DECRYPT_RAW(tbl.encrypted_col, tbl.key_col, tbl.iv_col, tbl.aad_col, 'AES-GCM', HEX_DECODE_BINARY('ff'));
BINARY;
# dialect: snowflake
TRY_DECRYPT(tbl.encrypted_col, 'passphrase');
BINARY;
# dialect: snowflake
TRY_DECRYPT(tbl.encrypted_col, 'passphrase', 'aad');
BINARY;
# dialect: snowflake
TRY_DECRYPT(tbl.encrypted_col, 'passphrase', 'aad', 'AES-GCM');
BINARY;
# dialect: snowflake
TRY_DECRYPT_RAW(tbl.encrypted_col, tbl.key_col, tbl.iv_col);
BINARY;
# dialect: snowflake
TRY_DECRYPT_RAW(tbl.encrypted_col, tbl.key_col, tbl.iv_col, tbl.aad_col);
BINARY;
# dialect: snowflake
TRY_DECRYPT_RAW(tbl.encrypted_col, tbl.key_col, tbl.iv_col, tbl.aad_col, 'AES-GCM');
BINARY;
# dialect: snowflake
TRY_DECRYPT_RAW(tbl.encrypted_col, tbl.key_col, tbl.iv_col, tbl.aad_col, 'AES-GCM', HEX_DECODE_BINARY('ff'));
BINARY;
--------------------------------------
-- T-SQL
--------------------------------------
@ -4659,3 +4959,15 @@ TIMESTAMPTZ;
# dialect: tsql
RADIANS(90);
INT;
--------------------------------------
-- MySQL
--------------------------------------
# dialect: mysql
LOCALTIME;
DATETIME;
# dialect: mysql
ELT(1, 'a', 'b');
VARCHAR;

93
tests/fixtures/optimizer/annotate_types.sql vendored
View file

@ -31,6 +31,15 @@ bool;
x is not null;
bool;
EXISTS(SELECT 1);
bool;
ALL(SELECT 1);
bool;
ANY(SELECT 1);
bool;
null;
UNKNOWN;
@ -145,3 +154,87 @@ ARRAY<FLOAT64>;
1 + (SELECT 2.5 AS c);
DOUBLE;
# dialect: bigquery
CASE WHEN TRUE THEN 2.5 ELSE CAST(3.5 AS BIGNUMERIC) END;
BIGNUMERIC;
# dialect: bigquery
CASE WHEN TRUE THEN CAST(3.5 AS BIGNUMERIC) ELSE 2.5 END;
BIGNUMERIC;
# dialect: bigquery
CASE WHEN TRUE THEN 3/10 ELSE CAST(3.5 AS BIGNUMERIC) END;
FLOAT64;
# dialect: bigquery
CASE WHEN TRUE THEN CAST(3.5 AS BIGNUMERIC) ELSE 3/10 END;
FLOAT64;
# dialect: bigquery
CASE WHEN TRUE THEN 2.4 ELSE 2.5 END;
FLOAT64;
# dialect: bigquery
CASE WHEN x < y THEN 3/10 WHEN x > y THEN 2 ELSE CAST(3.5 AS BIGNUMERIC) END;
FLOAT64;
# dialect: bigquery
CASE WHEN x < y THEN 2 WHEN x > y THEN 3/10 ELSE CAST(3.5 AS BIGNUMERIC) END;
FLOAT64;
# dialect: bigquery
CASE WHEN x < y THEN CAST(3.5 AS BIGNUMERIC) WHEN x > y THEN 3/10 ELSE 2 END;
FLOAT64;
# dialect: snowflake
BITSHIFTLEFT(255, 4);
INT;
# dialect: snowflake
BITSHIFTRIGHT(1024, 2);
INT;
# dialect: snowflake
BITSHIFTLEFT(X'FF', 4);
BINARY;
# dialect: snowflake
BITSHIFTRIGHT(X'FF', 4);
BINARY;
# dialect: snowflake
BITOR(BITSHIFTLEFT(5, 16), BITSHIFTLEFT(3, 8));
INT;
# dialect: snowflake
BITAND(BITSHIFTLEFT(255, 4), BITSHIFTLEFT(15, 2));
INT;
# dialect: bigquery
CAST(1 AS BIGNUMERIC) + 1.5;
BIGNUMERIC;
# dialect: bigquery
1.5 + CAST(1 AS BIGNUMERIC);
BIGNUMERIC;
# dialect: bigquery
1.5 + CAST(1 AS FLOAT64);
FLOAT64;
# dialect: bigquery
CAST(1 AS FLOAT64) + 1.5;
FLOAT64;
# dialect: bigquery
CAST(1 AS INT) + 1.5;
FLOAT64;
# dialect: bigquery
1.5 + CAST(1 AS INT);
FLOAT64;
# dialect: bigquery
IF(1 = 1, CAST(1 AS BIGNUMERIC) * 1.5, CAST(2 AS BIGNUMERIC));
BIGNUMERIC;

64
tests/test_dialect_entry_points.py Normal file
View file

@ -0,0 +1,64 @@
import unittest
from unittest.mock import Mock, patch
from sqlglot import Dialect
from sqlglot.dialects.dialect import Dialect as DialectBase
class FakeDialect(DialectBase):
pass
class TestDialectEntryPoints(unittest.TestCase):
def setUp(self):
Dialect._classes.clear()
def tearDown(self):
Dialect._classes.clear()
def test_entry_point_plugin_discovery_modern_api(self):
fake_entry_point = Mock()
fake_entry_point.name = "fakedb"
fake_entry_point.load.return_value = FakeDialect
mock_selectable = Mock()
mock_selectable.select.return_value = [fake_entry_point]
mock_entry_points = Mock(return_value=mock_selectable)
with patch("sqlglot.dialects.dialect.entry_points", mock_entry_points):
dialect = Dialect.get("fakedb")
self.assertIsNotNone(dialect)
self.assertEqual(dialect, FakeDialect)
fake_entry_point.load.assert_called_once()
mock_selectable.select.assert_called_once_with(group="sqlglot.dialects", name="fakedb")
def test_entry_point_plugin_discovery_legacy_api(self):
fake_entry_point = Mock()
fake_entry_point.name = "fakedb"
fake_entry_point.load.return_value = FakeDialect
mock_dict = Mock(spec=["get"])
mock_dict.get.return_value = [fake_entry_point]
mock_entry_points = Mock(return_value=mock_dict)
with patch("sqlglot.dialects.dialect.entry_points", mock_entry_points):
dialect = Dialect.get("fakedb")
self.assertIsNotNone(dialect)
self.assertEqual(dialect, FakeDialect)
fake_entry_point.load.assert_called_once()
mock_dict.get.assert_called_once_with("sqlglot.dialects", [])
def test_entry_point_plugin_not_found(self):
mock_selectable = Mock()
mock_selectable.select.return_value = []
mock_entry_points = Mock(return_value=mock_selectable)
with patch("sqlglot.dialects.dialect.entry_points", mock_entry_points):
dialect = Dialect.get("nonexistent")
self.assertIsNone(dialect)

7
tests/test_diff.py
View file

@ -354,5 +354,12 @@ class TestDiff(unittest.TestCase):
self._validate_delta_only(diff_delta_only(expr_src, expr_tgt), [])
def test_comments_do_not_affect_diff(self):
expr_src = parse_one("select a from tbl")
expr_tgt = parse_one("select a from tbl -- this is comment")
self.assertEqual(expr_tgt.args["from_"].this.comments, [" this is comment"])
self._validate_delta_only(diff_delta_only(expr_src, expr_tgt), [])
def _validate_delta_only(self, actual_delta, expected_delta):
self.assertEqual(set(actual_delta), set(expected_delta))

77
tests/test_generator.py
View file

@ -59,3 +59,80 @@ class TestGenerator(unittest.TestCase):
ast = parse_one(input_sql, read="postgres")
self.assertEqual(ast.sql(), expected_sql)
self.assertEqual(ast.sql("postgres"), expected_sql)
def test_pretty_nested_types(self):
def assert_pretty_nested(
datatype: exp.DataType,
single_line: str,
pretty: str,
max_text_width: int = 10,
**kwargs,
) -> None:
self.assertEqual(datatype.sql(), single_line)
self.assertEqual(
datatype.sql(pretty=True, max_text_width=max_text_width, **kwargs), pretty
)
# STRUCT
type_str = "STRUCT<a INT, b TEXT>"
assert_pretty_nested(
exp.DataType.build(type_str),
type_str,
"STRUCT<\n a INT,\n b TEXT\n>",
)
# STRUCT - type def shorter than max text width so stays one line
assert_pretty_nested(
exp.DataType.build(type_str),
type_str,
"STRUCT<a INT, b TEXT>",
max_text_width=50,
)
# STRUCT, leading_comma = True
assert_pretty_nested(
exp.DataType.build(type_str),
type_str,
"STRUCT<\n a INT\n , b TEXT\n>",
leading_comma=True,
)
# ARRAY
type_str = "ARRAY<DECIMAL(38, 9)>"
assert_pretty_nested(
exp.DataType.build(type_str),
type_str,
"ARRAY<\n DECIMAL(38, 9)\n>",
)
# ARRAY nested STRUCT
type_str = "ARRAY<STRUCT<a INT, b TEXT>>"
assert_pretty_nested(
exp.DataType.build(type_str),
type_str,
"ARRAY<\n STRUCT<\n a INT,\n b TEXT\n >\n>",
)
# RANGE
type_str = "RANGE<DECIMAL(38, 9)>"
assert_pretty_nested(
exp.DataType.build(type_str),
type_str,
"RANGE<\n DECIMAL(38, 9)\n>",
)
# LIST
type_str = "LIST<INT, INT, TEXT>"
assert_pretty_nested(
exp.DataType.build(type_str),
type_str,
"LIST<\n INT,\n INT,\n TEXT\n>",
)
# MAP
type_str = "MAP<INT, DECIMAL(38, 9)>"
assert_pretty_nested(
exp.DataType.build(type_str),
type_str,
"MAP<\n INT,\n DECIMAL(38, 9)\n>",
)

61
tests/test_optimizer.py
View file

@ -1070,6 +1070,7 @@ SELECT :with_,WITH :expressions,CTE :this,UNION :this,SELECT :expressions,1,:exp
"bignum_col": "BIGNUMERIC",
"date_col": "DATE",
"decfloat_col": "DECFLOAT",
"float_col": "FLOAT",
"timestamp_col": "TIMESTAMP",
"double_col": "DOUBLE",
"bigint_col": "BIGINT",
@ -1917,6 +1918,66 @@ SELECT :with_,WITH :expressions,CTE :this,UNION :this,SELECT :expressions,1,:exp
assert annotated.selects[0].type == exp.DataType.build("VARCHAR")
def test_bigquery_unnest_alias_shadowing(self):
"""Test that BigQuery UNNEST table alias shadows column names from other tables."""
sql = """
SELECT timeline_date
FROM UNNEST(GENERATE_DATE_ARRAY('2020-01-01', '2020-01-03')) AS timeline_date
LEFT JOIN production_tier ON production_tier.timeline_date = timeline_date
"""
schema = {"production_tier": {"timeline_date": "DATE", "id": "INT"}}
result = optimizer.qualify.qualify(
parse_one(sql, dialect="bigquery"),
schema=schema,
dialect="bigquery",
)
result_sql = result.sql(dialect="bigquery")
self.assertEqual(
result_sql,
"SELECT `timeline_date` AS `timeline_date` "
"FROM UNNEST(GENERATE_DATE_ARRAY('2020-01-01', '2020-01-03', INTERVAL '1' DAY)) AS `timeline_date` "
"LEFT JOIN `production_tier` AS `production_tier` "
"ON `production_tier`.`timeline_date` = `timeline_date`",
)
def test_struct_field_case_sensitivity_annotation(self):
schema = {"t": {"struct_col": "STRUCT<fooBar STRING>"}}
def _assert_dot_annotation(query: str, dialect: str, expected: exp.DataType.Type):
parsed = parse_one(query, dialect=dialect)
qualified = optimizer.qualify.qualify(parsed, schema=schema, dialect=dialect)
annotated = optimizer.annotate_types.annotate_types(
qualified, schema=schema, dialect=dialect
)
self.assertEqual(annotated.selects[0].type.this, expected)
# BigQuery is case-insensitive: exact field name match
_assert_dot_annotation(
"SELECT struct_col.fooBar FROM t", "bigquery", exp.DataType.Type.TEXT
)
# BigQuery: lower case
_assert_dot_annotation(
"SELECT struct_col.foobar FROM t", "bigquery", exp.DataType.Type.TEXT
)
# BigQuery: different case
_assert_dot_annotation(
"SELECT struct_col.Foobar FROM t", "bigquery", exp.DataType.Type.TEXT
)
# ClickHouse is case-sensitive: exact field name match
_assert_dot_annotation(
"SELECT struct_col.fooBar FROM t", "clickhouse", exp.DataType.Type.TEXT
)
# ClickHouse: lower case
_assert_dot_annotation(
"SELECT struct_col.foobar FROM t", "clickhouse", exp.DataType.Type.UNKNOWN
)
def test_annotate_object_construct(self):
sql = "SELECT OBJECT_CONSTRUCT('foo', 'bar', 'a b', 'c d') AS c"

30
tests/test_schema.py
View file

@ -241,6 +241,36 @@ class TestSchema(unittest.TestCase):
schema = MappingSchema(schema={"X": {"y": "int"}}, normalize=False, dialect="snowflake")
self.assertEqual(schema.column_names("x", normalize=True), ["y"])
# Check that identifiers in nested data types are normalized
# BigQuery: STRUCT field names should be lowercased
schema = MappingSchema({"t": {"col": "STRUCT<FooBar INT>"}}, dialect="bigquery")
col_type = schema.get_column_type("t", "col")
self.assertEqual(col_type.expressions[0].name, "foobar")
# Snowflake: STRUCT field names should be uppercased
schema = MappingSchema({"t": {"col": "STRUCT<FooBar INT>"}}, dialect="snowflake")
col_type = schema.get_column_type("T", "COL")
self.assertEqual(col_type.expressions[0].name, "FOOBAR")
# ClickHouse: STRUCT field names should preserve case (case-sensitive)
schema = MappingSchema({"t": {"col": "STRUCT<FooBar INT>"}}, dialect="clickhouse")
col_type = schema.get_column_type("t", "col")
self.assertEqual(col_type.expressions[0].name, "FooBar")
# Nested STRUCT field names should also be normalized
schema = MappingSchema(
{"t": {"col": "STRUCT<Outer STRUCT<Inner INT>>"}}, dialect="bigquery"
)
col_type = schema.get_column_type("t", "col")
self.assertEqual(col_type.expressions[0].name, "outer")
self.assertEqual(col_type.expressions[0].kind.expressions[0].name, "inner")
# ARRAY of STRUCT field names should also be normalized
schema = MappingSchema({"t": {"col": "ARRAY<STRUCT<FooBar INT>>"}}, dialect="bigquery")
col_type = schema.get_column_type("t", "col")
struct_type = col_type.expressions[0]
self.assertEqual(struct_type.expressions[0].name, "foobar")
def test_same_number_of_qualifiers(self):
schema = MappingSchema({"x": {"y": {"c1": "int"}}})