FrenchTech SAAS

Performance and scalability

• Indexing performance. Sphinx indexes up to 10-15 MB of text per second per single CPU core, that is 60+ MB/sec per server (on a dedicated indexing machine).
• Searching performance. Searching through 1,000,000-document, 1.2 GB text collection that we use for everyday development and testing runs at 500+ queries/sec on a 2-core desktop machine with 2 GB of RAM.
• Scalability. Biggest known Sphinx cluster indexes 25+ billion documents, resulting in over 9TB of data. Busiest known one is Craigslist, serving 300+ million search queries/day.

Key features

• Batch and Real-Time full-text indexes. Two index backends that support both efficient offline index construction andincremental on-the-fly index updates are available.
• Non-text attributes support. An arbitrary number of attributes (product IDs, company names, prices, etc) can be stored in the index and used either just for retrieveal (to avoid hitting the DB), or for efficient Sphinx-side search result set post-processing.
• SQL database indexing. Sphinx can directly access and index data stored in MySQL (all storage engines are supported), PostgreSQL, Oracle, Microsoft SQL Server, SQLite, Drizzle, and anything else that supports ODBC.
• Non-SQL storage indexing. Data can also be streamed to batch indexer in a simple XML format called XMLpipe, or inserted directly into an incremental RT index.
• Easy application integration. Sphinx comes with three different APIs, SphinxAPI, SphinxSE, and SphinxQL. SphinxAPI is a native library available for Java, PHP, Python, Perl, C, and other languages. SphinxSE, a pluggable storage engine for MySQL, enables huge result sets to be shipped directly to MySQL server for post-processing. SphinxQL lets the application query Sphinx using standard MySQL client libary and query syntax.
• Advanced full-text searching syntax. Our querying engine supports arbitrarily complex queries combining boolean operators, phrase, proximity, strict order, and quorum matching, field and position limits, exact keyword form matching, substring searches, etc.
• Rich database-like querying features. Sphinx does not limit you to just keyword searching. On top of full-text search result set, you can compute arbitrary arithmetic expressions, add WHERE conditions, do ORDER BY, GROUP BY, use MIN/MAX/AVG/SUM, aggregates etc. Essentially, full-blown SQL SELECT is supported.
• Better relevance ranking. Unlike many other engines, Sphinx does not solely rely on 30-year-old statistical ranking that only considers keyword frequencies, nor limits you to it. By default, Sphinx additionally analyzes keyword proximity, and ranks closer phrase matches higher, with perfect matches ranked on top. Also, ranking is flexible: you can choose from a number of built-in relevance functions, tweak their weights by using expressions, or develop new ones.
• Flexible text processing. Sphinx indexing features include full support for SBCS and UTF-8 encodings (meaning that effectively all world's languages are supported); stopword removal and optional hit position removal (hitless indexing); morphology and synonym processing through word forms dictionaries and stemmers; exceptions and blended characters; and many more.
• Distributed searching. Searches can be distributed across multiple machines, enabling horizontal scale-out and HA (High Availability).

Database Testing:

What is Database Testing?

• In Database testing backend records are tested which have been inserted through the web or desktop applications. The data which is displaying in the web application should match with the data stored in the Database.

To perform the Database testing, the tester should be aware of the below mentioned points:

• The tester should understand the functional requirements, business logic, application flow and database design thoroughly.
• The tester should figure out the tables, triggers, store procedures, views and cursors used for the application.
• The tester should understand the logic of the triggers, store procedures, views and cursors created.
• The tester should figure out the tables which get affected when insert update and delete (DML) operations are performed through the web or desktop applications.

With the help of the above mentioned points, the tester can easily write the test scenarios for Database testing.

Example Test Cases for Database Testing:

• Verify the database name: The database name should match with the specifications.
• Verify the Tables, columns, column types and defaults: All things should match with the specifications.
• Verify whether the column allows a null or not.
• Verify the Primary and foreign key of each table.
• Verify the Stored Procedure:
• Test whether the Stored procedure is installed or not.
• Verify the Stored procedure name
• Verify the parameter names, types and number of parameters.
• Test the parameters if they are required or not.
• Test the stored procedure by deleting some parameters
• Test when the output is zero, the zero records should be affected.
• Test the stored procedure by writing simple SQL queries.
• Test whether the stored procedure returns the values
• Test the stored procedure with sample input data.
• Verify the behavior of each flag in the table.
• Verify the data gets properly saved into the database after the each page submission.
• Verify the data if the DML (Update, delete and insert) operations are performed.
• Check the length of every field: The field length in the back end and front end must be same.
• Verify the database names of QA, UAT and production. The names should be unique.
• Verify the encrypted data in the database.
• Verify the database size. Also test the response time of each query executed.
• Verify the data displayed on the front end and make sure it is same in the back end.
• Verify the data validity by inserting the invalid data in the database.
• Verify the Triggers.

These are some general step by step techniques for database scaling. Although most of the engineers don’t get enough chance to implement these techniques, but as a whole it’s better to get a broader idea about such system which in future may help you to do better system & architecture designing.

In my next articles, I will try to discuss some of the concepts in details.

For nearly a decade, Hadoop was the poster child for “big data.” It was new, it was open source, it launched an entire market of products and vendors, and it was inspired by — and in many cases, was — the technology behind the world’s largest websites. However, looking back with 20/20 hindsight, it seems clear that Hadoop was never going to live up to its lofty expectations. It was big and new, but released into a world that would soon come to value speed, flexibility, micro, and even known quantities.

My sample Spring Boot app spring-elastic-genie has a REST API that looks for movies in OMDB and saves the results in PostgreSQL. These results then get indexed by Elasticsearch at which point you can visualize the results in Kibana. You should know in advance that this is not a how-to-code tutorial but a high-level architecture overview of using Spring Boot with ELK on-premise, in Docker.

Pros of Scaling-out

• Much cheaper compared to scaling-up
• Takes advantage of smaller systems
• Easy to upgrade
• Resilience is improved due to the presence of discrete, multiple systems
• Easier to run fault-tolerance
• Supporting linear increases in capacity

Cons of Scaling-out

• The licensing fees are more
• Utility costs such as cooling and electricity are high
• It has a bigger footprint in the Data Center
• More networking equipment such as routers and switches may be needed